Base Models Can Reason By Taking a Cue From Training Data

Sophie L. Wang, Amil Dravid, Rulin Shao, Kevin Farhat, Sewon Min, Alexei A. EfrosarXiv 2026 · arXiv

TL;DR

This paper reveals that base LLMs often possess latent reasoning capabilities that are only triggered by specific "token cues" at the start of a response. By simply prefilling a base model with cues like .\n\n Okay, the authors demonstrate that base models can match or even exceed the performance of their reinforcement learning (RL)-trained counterparts, suggesting that RL primarily functions by up-weighting these learned associations rather than teaching new reasoning skills.

What Problem Does This Paper Address?

A central debate in AI is whether reasoning "emerges" during post-training (like RLHF or RL-Zero) or if the capabilities are already present in base models, merely waiting to be unlocked. While RL-trained models like DeepSeek-R1 show impressive "aha moments" and self-correction, it remains unclear if RL is teaching the model how to reason or simply when to invoke reasoning behaviors it learned during pre-training.

The authors investigate the causal link between the very first tokens a model generates and the complex reasoning behavior that follows. They aim to understand how training data creates these associations and how much of the "RL gain" is actually just a shift in the probability of generating these crucial starting cues.

Methods

The researchers employed several novel techniques to dissect this behavior:

  1. Cue Discovery: To find effective cues without using reference answers, they used beam search to identify high-probability response openings. They then selected the cue that resulted in the highest "answer agreement" (lowest Shannon entropy) among multiple sampled completions.
  2. Prefilling: Instead of letting the model generate the start of the response, they "prefilled" the context with specific tokens (e.g., .\n\n Okay) and observed the resulting continuation.
  3. Causal Data Interventions: To prove the cues are learned from specific data, they retrained models on edited versions of their mid-training data. For example, they replaced every instance of "okay" with "chicken" to see if "chicken" would become a reasoning trigger.
  4. Representational Analysis: They compared the hidden states of the model during cued responses to the hidden states induced by different types of training data (e.g., synthetic reasoning traces vs. short Q&A pairs).
  5. Targeted Weight Editing: They developed a method to edit a single weight matrix in the final MLP layer to increase the probability of a specific cue, testing if this "shortcut" could recover RL-level performance.

Datasets

The study primarily utilized the Olmo-3-7B and Qwen3-14B model families.

  • Training/Intervention: They used the 10B and 100B token mid-training mixtures for Olmo-3 and the final decay stage data for SmolLM3-3B.
  • Evaluation: Reasoning was tested on MATH-500, GSM8K, AMC 23, AIME 2024, and HumanEval (coding).
  • Safety: The authors used XSTest to evaluate refusal and compliance behaviors.

Key Insights and Ideas

  • The Pavlovian LLM: Reasoning in LLMs acts like a conditioned response. Just as Pavlov's dogs salivated at a bell, LLMs trigger complex reasoning traces when they see specific, seemingly uninformative tokens like "Okay" that were associated with reasoning traces in their training data.
  • RL as a Probability Shifter: The authors found that the primary difference between a base model and an RL-trained model is concentrated in the first two tokens. RL doesn't necessarily improve the "reasoning engine"; it mostly increases the likelihood that the model will hit the "ignition switch" (the cue) for reasoning.
  • Arbitrary Cues: Reasoning triggers are not semantic. By editing the training data, the authors made "chicken" a reasoning cue and made the instruction "Think duck duck goose" as effective as "Think step by step" (see Figure 7 in the paper).
  • Safety is Cue-Dependent: The same base model can be made to either refuse a harmful request or comply with it simply by changing the opening tokens (e.g., I'm sorry vs Okay ,), highlighting that "alignment" is often a matter of steering the model toward specific learned data distributions.
Figure 1. A base model’s first tokens steer reasoning behavior through associations learned from training data. (Top) Olmo-3-7B’s next-token probabilities are conditioned on the prompt and preceding tokens. Starting with .\n Answer leads to a short answer, whereas .\n\n Okay leads to deliberation and reasoning performance comparable to RL (Sec. 2.2). RL increases the probability of the reasoning cue .\n\n Okay (Sec. 2.3). Replacing “okay” with “chicken” in mid-training data makes .\n\n Chicken elicit reasoning too (Sec. 3.1). Representational analysis connects these behaviors to training data (Sec. 3.2). (Bottom) MATH-500 pass@1 for each opening on the base model, after RL, and after the data edit on the 100B-token mid-training mix.

Key Results

  • Matching RL Performance: Prefilling Olmo-3-7B with .\n\n Okay raised its MATH-500 accuracy from 42% to 78%, surpassing its RL-trained version (75%). Similarly, Qwen3-14B jumped from 72% to 87% with the cue Alright ,.
Figure 2. Main results showing that fixing the first two tokens brings base models to RL-level accuracy across multiple math and coding benchmarks.
  • Causal Link: In the "rename" experiment, the .\n\n Chicken cue raised MATH-500 accuracy from 2.4% to 37.2%, proving that the behavior is directly tied to the associations in the training mix.
Figure 5. Data edits demonstrate that replacing "Okay" with "Chicken" in training data successfully transfers the reasoning-trigger effect to the new word.
  • Efficiency: Prefilling the cue before any RL updates matches the accuracy that standard RL takes roughly 100 steps to achieve.

Limitations and Future Work

  • Model Dependence: The effectiveness of specific cues is highly dependent on the model's specific training data. For instance, the authors found no effective cues for Llama-3.1-8B, likely because its training data structure differs significantly from Olmo or Qwen.
  • R1-Zero Focus: The study focuses on RL applied directly to base models. It is possible that complex, multi-stage post-training pipelines (SFT + RLHF) introduce capabilities that cannot be elicited by simple cues alone.
  • Future Directions: The authors suggest re-examining high-level semantic instructions. If "Think step by step" is just a learned association rather than a semantic command, we may need new ways to interpret how models follow instructions. They also propose "designing" training data to intentionally pair useful capabilities with easy-to-trigger cues.

Summary updated

Abstract

In this paper, we study how training data creates associations between the tokens at the start of a base model's response and the reasoning behavior that follows. First, we demonstrate that fixing particular starting token cues makes a base model's performance competitive with that of its reinforcement learning (RL)-trained counterparts on math and coding. For instance, the cue ".\n\nOkay" raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while "Alright," raises Qwen3-14B's from 72% to 87%. Second, RL makes these cues more likely, while fixing them recovers much of its performance gain over the base model. Third, we trace the reasoning effects of token cues to the training data. We perform causal data interventions to turn an arbitrary word, such as "chicken", into an effective reasoning cue, or remove an existing cue's effect. A similar edit makes the prompt instruction "Think duck duck goose" as effective as "Think step by step" at eliciting reasoning. We also find that the hidden state representations induced by different cues correlate with different document types from the training set. Finally, we extend our study of token cues with a case study in language model safety, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.
BibTeX
@misc{wang2026base,
  author = {Sophie L. Wang and Amil Dravid and Rulin Shao and Kevin Farhat and Sewon Min and Alexei A. Efros},
  title = {{Base Models Can Reason By Taking a Cue From Training Data}},
  year = {2026},
  eprint = {2610.06851},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.06851},
}
Figure