TL;DR
This paper reveals that base LLMs often possess latent reasoning capabilities that are only triggered by specific "token cues" at the start of a response. By simply prefilling a base model with cues like .\n\n Okay, the authors demonstrate that base models can match or even exceed the performance of their reinforcement learning (RL)-trained counterparts, suggesting that RL primarily functions by up-weighting these learned associations rather than teaching new reasoning skills.
What Problem Does This Paper Address?
A central debate in AI is whether reasoning "emerges" during post-training (like RLHF or RL-Zero) or if the capabilities are already present in base models, merely waiting to be unlocked. While RL-trained models like DeepSeek-R1 show impressive "aha moments" and self-correction, it remains unclear if RL is teaching the model how to reason or simply when to invoke reasoning behaviors it learned during pre-training.
The authors investigate the causal link between the very first tokens a model generates and the complex reasoning behavior that follows. They aim to understand how training data creates these associations and how much of the "RL gain" is actually just a shift in the probability of generating these crucial starting cues.
Methods
The researchers employed several novel techniques to dissect this behavior:
- Cue Discovery: To find effective cues without using reference answers, they used beam search to identify high-probability response openings. They then selected the cue that resulted in the highest "answer agreement" (lowest Shannon entropy) among multiple sampled completions.
- Prefilling: Instead of letting the model generate the start of the response, they "prefilled" the context with specific tokens (e.g.,
.\n\n Okay) and observed the resulting continuation. - Causal Data Interventions: To prove the cues are learned from specific data, they retrained models on edited versions of their mid-training data. For example, they replaced every instance of "okay" with "chicken" to see if "chicken" would become a reasoning trigger.
- Representational Analysis: They compared the hidden states of the model during cued responses to the hidden states induced by different types of training data (e.g., synthetic reasoning traces vs. short Q&A pairs).
- Targeted Weight Editing: They developed a method to edit a single weight matrix in the final MLP layer to increase the probability of a specific cue, testing if this "shortcut" could recover RL-level performance.
Datasets
The study primarily utilized the Olmo-3-7B and Qwen3-14B model families.
- Training/Intervention: They used the 10B and 100B token mid-training mixtures for Olmo-3 and the final decay stage data for SmolLM3-3B.
- Evaluation: Reasoning was tested on MATH-500, GSM8K, AMC 23, AIME 2024, and HumanEval (coding).
- Safety: The authors used XSTest to evaluate refusal and compliance behaviors.
Key Insights and Ideas
- The Pavlovian LLM: Reasoning in LLMs acts like a conditioned response. Just as Pavlov's dogs salivated at a bell, LLMs trigger complex reasoning traces when they see specific, seemingly uninformative tokens like "Okay" that were associated with reasoning traces in their training data.
- RL as a Probability Shifter: The authors found that the primary difference between a base model and an RL-trained model is concentrated in the first two tokens. RL doesn't necessarily improve the "reasoning engine"; it mostly increases the likelihood that the model will hit the "ignition switch" (the cue) for reasoning.
- Arbitrary Cues: Reasoning triggers are not semantic. By editing the training data, the authors made "chicken" a reasoning cue and made the instruction "Think duck duck goose" as effective as "Think step by step" (see Figure 7 in the paper).
- Safety is Cue-Dependent: The same base model can be made to either refuse a harmful request or comply with it simply by changing the opening tokens (e.g.,
I'm sorryvsOkay ,), highlighting that "alignment" is often a matter of steering the model toward specific learned data distributions.
Key Results
- Matching RL Performance: Prefilling Olmo-3-7B with
.\n\n Okayraised its MATH-500 accuracy from 42% to 78%, surpassing its RL-trained version (75%). Similarly, Qwen3-14B jumped from 72% to 87% with the cueAlright ,.
- Causal Link: In the "rename" experiment, the
.\n\n Chickencue raised MATH-500 accuracy from 2.4% to 37.2%, proving that the behavior is directly tied to the associations in the training mix.
- Efficiency: Prefilling the cue before any RL updates matches the accuracy that standard RL takes roughly 100 steps to achieve.
Limitations and Future Work
- Model Dependence: The effectiveness of specific cues is highly dependent on the model's specific training data. For instance, the authors found no effective cues for Llama-3.1-8B, likely because its training data structure differs significantly from Olmo or Qwen.
- R1-Zero Focus: The study focuses on RL applied directly to base models. It is possible that complex, multi-stage post-training pipelines (SFT + RLHF) introduce capabilities that cannot be elicited by simple cues alone.
- Future Directions: The authors suggest re-examining high-level semantic instructions. If "Think step by step" is just a learned association rather than a semantic command, we may need new ways to interpret how models follow instructions. They also propose "designing" training data to intentionally pair useful capabilities with easy-to-trigger cues.
Summary updated