Study finds LLMs keep reasoning even with thinking mode switched off

What did the Thinking Inertia study find?

The paper, “Thinking Inertia: LLMs Keep Thinking When Told Not To”, was posted to arXiv on 8 October 2026 by Dianqiao Lei of Tsinghua University, Kevin Qinghong Lin and Philip Torr of the University of Oxford, and Pan Lu and James Zou of Stanford University. Lin is the project lead. The authors’ GitHub README says the work has been accepted as a poster at NeurIPS 2026.

Its central claim is in the title. Many models now ship with a thinking mode and a way to turn it off, and developers use the off setting to cut latency and token cost. The authors find that switching thinking off, or adding an instruction such as “Give the final answer directly without any reasoning”, often leaves the model writing out a derivation before its answer anyway. They call this persistence “thinking inertia”.

The pattern follows the shape of the answer. Visible reasoning was rarest on yes/no questions, more common on multiple choice, and almost universal on open-ended maths. The evaluation code is on GitHub under the MIT licence, the model outputs are published as a Hugging Face dataset, and the paper itself is released under CC0.

Why is a think-off switch a poor test of no-thinking?

The authors argue that the usual ways of checking whether a model skipped reasoning are proxies that can fail. One is the prompt or API setting itself: the model was run with thinking disabled, so it is assumed not to have reasoned. Another is format: no separate reasoning block or think tag appeared. A third is length: the output was short.

Each can mislead. With enable_thinking set to false, a model asked “If 3x + 1 = 10, what is x?” can still reply “Subtract 1 from both sides: 3x = 9. Then divide by 3, so x = 3.” A reply with no think tag can contain the same steps in ordinary prose. And a very short reply such as “3x = 9, so x = 3” still exposes the key inferential step. Conversely, a long reply may be filler, such as “Let me think about this. The answer should be…”, with no real inference in it.

This matters for anyone who treats a model’s thinking and non-thinking modes as a clean switch between two behaviours. The paper limits its claims to what is visible in the output; it does not say anything about computation hidden inside the model.

How did the researchers measure no-thinking?

Instead of trusting the setting, the authors split every response into the pre-answer text and the final answer, then score the pre-answer text at three levels.

The first is the Empty-Thinking Rate (ETR): the share of responses that are strictly answer-only, with nothing but the permitted answer wrapper. The second is Question–Pre-answer Relevance (QRel), a similarity score between the question and the pre-answer text computed with the instruction-aware Qwen3-Embedding-4B model; it measures relevance, not reasoning. The third is the Explicit Inference Rate (EIR): a blinded GPT-5.5 judge, run at temperature 0, labels each non-empty pre-answer text as generic, paraphrase, relevant but non-inferential, explicit reasoning or unclear, and only explicit reasoning counts.

To check the metrics, five human annotators reviewed a 600-output sample. The strict answer-only check matched manual inspection on 565 of 600 outputs, QRel correlated with human relevance judgments at a Spearman coefficient of 0.829, and human reasoning labels agreed with the GPT-5.5 judge 98.8% of the time and with an independent Claude Opus 4.8 judge 96.7% of the time.

Which models, benchmarks and prompts were tested?

The study covers six models that expose an explicit thinking control: Qwen3-4B, Qwen3-32B and Qwen3-235B-A22B, DeepSeek-V4-Flash (284B), OpenAI’s GPT-5.4, and a preview of Google’s Gemini 3 Flash. Gemini’s lowest setting is “minimal” thinking, which the authors treat as near-off rather than fully off. Each was run on two benchmarks in each of three answer formats: BoolQ and StrategyQA for yes/no, MMLU and MMLU-Pro for multiple choice, and GSM8K and MATH for open-ended answers. The full corpus contains 572,688 outputs.

Every question was run under six modes. Two encourage reasoning and serve as references: thinking on, and thinking off plus “Think step by step”. The other four form the no-thinking space: thinking off with no extra instruction (the native baseline), a soft prompt allowing a short lead-in but no explanation, a strict prompt demanding the answer without any reasoning, and prefix forcing, which pre-fills “The answer is \boxed{” so the model starts mid-answer.

What happens when you force a direct answer?

With thinking natively switched off, most models produced answer-only replies on 80–100% of yes/no questions. On multiple choice the rate varied from near zero to 95% depending on the model. On open-ended questions it collapsed to nearly zero for every model, and almost every reply carried visible explicit reasoning. DeepSeek-V4-Flash, for example, kept visible explicit inference in 99.9% of open-ended responses with thinking disabled.

Stronger instructions helped, at a cost. Across the five models with a fully disabled thinking setting, moving from the native think-off mode to the strict “no reasoning” prompt raised the mean answer-only rate on open-ended tasks from near zero to roughly 40%, and explicit inference across the corpus fell from 99.69% to 62.00%, but mean accuracy dropped by about 15 percentage points. On yes/no and multiple-choice tasks, the strict prompt cut explicit inference (from 5.15% to 0.12% and from 45.57% to 25.39%) while accuracy stayed roughly flat; yes/no accuracy sat around 85% in every no-thinking mode.

Prefix forcing, which might look like the strongest control, did not work as expected. On multiple choice and open-ended questions the answer-only rate fell back towards zero and reasoning rebounded, because models carried on writing derivations after the pre-filled opening. In the reverse direction, a plain “Think step by step” prompt brought visible reasoning back even with the thinking switch off. GPT-5.4 was the main exception, staying largely answer-only on yes/no and multiple-choice questions.

Why does the answer format matter so much?

To separate format from difficulty, the authors took the same numeric maths questions from MMLU and MMLU-Pro and rewrote them three ways, then ran Qwen3-32B with thinking off. As a yes/no check of a proposed answer, accuracy was 78.9% and 62.5% of replies were answer-only. In the original multiple-choice form, accuracy was 77.8% and only 10.5% were answer-only. As open-ended questions, accuracy fell to 52.6% and no reply was answer-only, with explicit reasoning in 90.14% of outputs.

The explanation the authors offer is that a candidate answer gives the model something to verify or select, while an open question forces it to construct the value, which seems to need visible intermediate steps. Subject also matters: within MMLU, STEM questions had both the highest accuracy and the lowest answer-only rates, 0.3–36.4% depending on the model, so a model finding a subject easy does not mean it will answer it directly.

What should developers using no-thinking modes do?

The practical lesson is to measure rather than assume. If a product relies on a no-thinking mode to keep responses fast or cheap, the paper suggests checking the actual outputs for hidden derivations, especially on free-form questions, rather than trusting the API flag or a prompt instruction. Parsers that expect only an answer should also be ready for extra text.

Where a direct answer really matters, giving the model a constrained answer space appears to help. That is the same intuition behind decision models that score a fixed set of options instead of generating free text. For open-ended tasks, the accuracy drop suggests that forcing silence can simply make answers worse, so a short reasoning budget may be a better trade than an answer-only rule, an approach familiar from reasoning models such as DeepSeek-R1.

This is AI Lookout’s reading of the results, not a recommendation made in the paper, which frames no-thinking as a capability that should be evaluated alongside reasoning ability.

What are the limits of the study?

The study measures only visible text. A model that answers in one token may still have done the work internally, and one that writes a derivation may not have used it; the authors say explicitly that they make no claims about latent computation.

The coverage is also narrow: six models, six English benchmarks, deterministic decoding where supported, and single-turn questions. Multilingual, coding, tool-use, multi-turn and sampled settings are untested, and proprietary endpoints such as GPT-5.4 and Gemini 3 Flash can change behaviour over time. The reasoning metric depends on an LLM judge and its rubric, although the second judge and the human study support it. AI Lookout is not aware of an independent reproduction yet.