The "aha moment" has become a central trope in the narrative of modern reasoning models. We’ve all seen the traces: a model starts solving a math problem, stops, writes "Wait, is that correct?", and then corrects its own logic. Conventional wisdom suggests that Reinforcement Learning (RL) "discovers" this reflective behavior, coining these markers as it learns to maximize rewards.
A new paper from researchers at HKUST, Peking University, and Huawei Noah’s Ark Lab challenges this narrative. Their finding? The word "wait" is inherited, not invented.
TL;DR
Reinforcement Learning (RL) doesn't teach models to say "wait"; it merely amplifies and reshapes markers already present in pre-training or fine-tuning data. By tracing the "provenance" of these markers across 120 model checkpoints, the authors show that models can achieve massive reasoning gains without ever using the word, debunking the idea that RL "coins" reflective language.
What Problem Does This Paper Address?
As models like DeepSeek-R1-Zero popularized the "aha moment," researchers began using keywords like "wait" or "recheck" as proxies for measuring a model's reasoning ability. However, it was unclear whether RL was actually teaching the model the concept of doubt or if the model was simply latching onto high-reward linguistic patterns it had already seen.
The authors set out to determine the true origin of these revision markers. Does RL "invent" them from scratch (the "common view"), or are they artifacts of the training data?
Methods
The researchers treated the training history of open models as a diachronic corpus—a collection of texts that changes over time. They analyzed three classes of model lineages:
- Fully open lineages: Tracking OLMo-3 from pre-training through mid-training to RL.
- Zero-RL runs: Training models like Qwen2.5 and Mistral from "marker-free" bases using only outcome rewards.
- Fine-tuned descendants: Models trained on reasoning traces (like R1-Distill).
To measure the evolution of "wait," they developed several label-free linguistic indices:
- Frequency: How often the word appears.
- Fixation: Whether it moves to the start of sentences.
- Attrition: Whether the form "wears down" (e.g., dropping the comma in "Wait,").
- Injected-Error Alarm Test: A clever probe where they intentionally insert an arithmetic error into a correct reasoning chain and see if the model triggers a "wait" marker.
Datasets
The study primarily used MATH-500 for evaluation, supplemented by AIME 2024/2025, AMC23, GSM8K, and MMLU-Pro. They also released a "provenance map" of 120 model endpoints and expert-consensus audits of reasoning traces.
Key Insights and Ideas
- RL is not a linguist: In 17 out of 18 Zero-RL runs from bases that lacked the word "wait," the models achieved massive accuracy gains (median +42.9 points) while the word "wait" remained virtually absent. RL doesn't need the word to learn the logic.
- The "Wait" comes from the data: In the OLMo-3 lineage, "wait" didn't appear during RL; it entered during the mid-training phase when reasoning-rich data was introduced.
- Linguistic Attrition: When RL does have access to the marker, it "wears it down." In OLMo-3 RL-Zero, the canonical "Wait," form dropped from 89% to 42%, often becoming a repetitive "wait, wait" chain.
- The Liberal Alarm: As RL progresses, the "wait" marker becomes a more sensitive but "liberal" alarm. It fires more reliably when there is an actual error, but it also starts firing on perfectly correct steps (false alarms).
Key Results
- Zero-RL rarely coins markers: Models trained from scratch with RL on math problems almost never start saying "wait" if it wasn't in their base weights.
- Data is the seed: 13 out of 14 models fine-tuned on reasoning traces adopted the marker from their training data.
- Recipe matters: Different RL algorithms treat the marker differently. While standard RL-Zero recipes multiply the marker, a KL-free GRPO recipe actually erased a marker that had been seeded by fine-tuning.
- Judge Unreliability: The authors found that "LLM-as-a-judge" metrics for reasoning can actually reverse human-labeled trends, over-calling revisions in base models and under-calling them in RL models.
Limitations and Future Work
The study is primarily observational and focused on English-language mathematics. While it provides a robust framework for tracking lexical markers, it doesn't fully capture "silent" reflection—where a model might be reconsidering its steps without explicitly writing a marker.
Future work could extend this "provenance tracking" to other domains (like coding) or other languages to see if the "inherited, not invented" rule holds globally across the LLM ecosystem. The authors also suggest that monitor-builders should calibrate their tools to specific model checkpoints rather than treating "wait" as a universal signal of doubt.
Summary updated