Thought without systematicity? Evaluating reasoning models on rule induction tasks

Papers

arxiv:2609.13948

Published on Sep 12

· Submitted by

Simon Schug on Sep 15

· Princeton University

Upvote

3

Authors:

Simon Schug ,

Abstract

Reasoning models frequently fail on structurally equivalent task variants, indicating a lack of systematicity in their cognitive abilities.

Generated by thinkingmachines/Inkling-Small

A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same task. Here, we extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models. Each task family has compositional structure that we use to create structurally equivalent task variations through task isomorphisms such as recombination and substitution. We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task. These findings suggest that many model behaviors lack systematicity, rendering it difficult to robustly establish the cognitive abilities of reasoning models beyond the particular contexts they were evaluated in.

View arXiv page View PDF GitHub 1 Add to collection

Community

smonsays

Paper author Paper submitter 7 days ago

Do reasoning models have systematicity of thought?

librarian-bot

7 days ago

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• Thinking effort aligns between humans and reasoning models in abductive reasoning (2026)

• Not All LLM Reasoning is Visible in the Chain-of-Thought (2026)

• Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs (2026)

• Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B (2026)

• Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning (2026)

• Induction and Inquiry via Probabilistic Reasoning over Language and Code (2026)

• Be Consistent! Enhancing Robust Visual Reasoning in LVLMs with Consistency Constraints (2026)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on HF Mirror checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Upvote

3

Get this paper in your agent:

hf papers read 2609.13948

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2609.13948 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2609.13948 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2609.13948 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.