When a large language model (LLM) fails a complex logic puzzle, our default assumption is that it lacks the necessary cognitive machinery. We assume it must be fine-tuned on thousands of step-by-step examples or trained via massive reinforcement learning to "learn" how to think.
However, a growing body of research suggests a more surprising reality: the capacity for complex, multi-step reasoning is already latent within the pre-trained weights of base models. The failure to reason is often not a lack of capability, but an execution or activation failure.
During pre-training on massive web-scale corpora, models internalize highly structured representations of logic, code, mathematical proofs, and causal arguments. But standard inference methods—like greedy decoding—tend to collapse these rich, multi-step pathways in favor of short, high-probability direct answers. Unlocking "System 2" thinking is therefore less about teaching new skills and more about navigating the model into its existing, high-performance reasoning states.
Linguistic Triggers: Scaffolding the Output Space
The simplest evidence for latent reasoning comes from prompt engineering. In their seminal work, Shixiang (Shane) Gu et al. demonstrated that appending a simple phrase like "Let's think step by step" triggers zero-shot chain-of-thought (CoT) reasoning, drastically improving performance without any task-specific examples (Large Language Models are Zero-Shot Reasoners).
But why does a simple linguistic nudge unlock such deep capabilities? Boshi Wang et al. discovered that the actual logic provided in few-shot prompts matters surprisingly little; even prompts containing invalid reasoning steps achieve 80-90% of the performance of valid ones (Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters). This indicates that prompts act primarily as a format trigger or structural scaffold. They signal the model to activate its inherent sequential processing capabilities rather than teaching it new logical rules. As Yejin Bang et al. note, a model's failure on a complex task often stems from a lack of task-specific activation rather than an inherent absence of the underlying cognitive ability (A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity).
To maximize this activation, researchers design explicit structural templates that force the model to bypass superficial shortcuts. For instance, Lei Wang et al. introduced "Plan-and-Solve" prompting to scaffold task decomposition (Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models), while Cong Liu et al. leveraged human-inspired metacognitive methodologies to scale test-time computation without any training (Chain of Methodologies: Scaling Test Time Computation without Training). Similarly, Brown Ebouky et al. showed that providing a structured interface for internal operations like backtracking and self-reflection can unlock these latent abilities to match or exceed models trained with heavy post-training (Eliciting Reasoning in Language Models with Cognitive Tools).
Decoding and Sampling: Uncovering Hidden Trajectories
If reasoning is already inside the model, why doesn't it show up by default? The answer lies in how we sample tokens. Standard greedy decoding selects the single most likely token at each step, which naturally biases the model toward short, direct answers (e.g., jumping straight to an incorrect final answer).
Xuezhi Wang et al. showed that by exploring alternative decoding paths (looking at the top-$k$ most likely first tokens), valid chain-of-thought reasoning paths can be elicited entirely without explicit prompts (Chain-of-Thought Reasoning Without Prompting). The reasoning paths are already present in the model's probability distribution; standard decoding simply suppresses them. This phenomenon extends beyond text: Peter West et al. demonstrated that standard autoregressive models possess latent bidirectional reasoning capabilities that can be activated purely through specialized decoding strategies (Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models).
Standard Greedy Decoding:
Prompt ──> [High-prob Token] ──> Direct (Often Incorrect) Answer
Latent Activation Decoding:
Prompt ───┬─> [Path A] ───> [Internal Reflection] ───> Correct Answer
├─> [Path B] ───> [Self-Correction] ─────> Correct Answer
└─> [Path C] ───> [Greedy Path] ──────────> Direct Answer
By shifting from greedy decoding to training-free Markov Chain Monte Carlo (MCMC) "Power Sampling," Aayush Karan et al. demonstrated that base models can achieve reasoning performance that matches or exceeds state-of-the-art reinforcement learning methods (Reasoning with Sampling: Your Base Model is Smarter Than You Think). This strongly implies that reinforcement learning primarily serves to surface existing high-quality reasoning paths rather than instilling entirely new cognitive behaviors.
Representation Engineering: Steering Internal Hidden States
Instead of modifying the text input or the token selection process, representation engineering directly manipulates the model's internal activations. This approach treats latent reasoning as specific coordinates or directions within the model's hidden activation space.
Xinyu Tang et al. introduced GLoRE, a training-free method that steers the model's hidden states into high-performance, long-form reasoning modes across math and physics (Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering). In a similar vein, the authors of Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization utilized gradient-based optimization to navigate the hidden representation space, balancing reasoning likelihood with linguistic fluency to unlock multi-step logic without fine-tuning.
This internal manipulation can target incredibly precise mechanisms:
- Activation Control: Zekai Zhao et al. found that identifying and amplifying a tiny set of specific neural activations in the model's final layers can elicit long-form reasoning and self-correction (Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models).
- Linear Feature Toggles: Yihuai Hong et al. identified "Linear Reasoning Features" (LiReFs)—single directions in the activation space that act as a toggle between rote memorization and systematic reasoning (The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction). By manually steering along these directions, they can nudge the model to apply systematic logic to novel problems.
Minimalist Optimization: Igniting the Spark
If massive datasets aren't required to build reasoning capabilities, how much data or optimization is actually needed to activate them? Recent work shows that the threshold is shockingly low because the underlying knowledge redundancy is so high.
Shivam Agarwal et al. demonstrated that a simple objective like Entropy Minimization (EM)—which merely forces a model to be more confident and commit to a deterministic reasoning path—can unlock sophisticated problem-solving skills without any labeled data at all (The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning).
When supervision is used, it often acts as an alignment trigger rather than an instructional teacher. For instance, Yiping Wang et al. showed that reinforcement learning with a verifiable reward can dramatically boost mathematical reasoning using just one training example, revealing a "post-saturation generalization" effect where the latent logic of the base model is fully ignited (Reinforcement Learning for Reasoning in Large Language Models with One Training Example). Similarly, Ruimeng Ye et al. demonstrated that weak or noisy supervision can act as an effective trigger to align a model's existing reasoning skills to solve complex, multi-step problems (Weak-to-Strong Generalization with Failure Trajectories).
Ultimately, this paradigm shifts our understanding of machine intelligence. We are moving away from the idea that models must be explicitly taught how to think step-by-step. Instead, the evidence shows that the pre-training process creates a vast reservoir of latent logical pathways. The challenge of modern AI design is increasingly focused on building the right keys—whether through prompts, decoding algorithms, or representation steering—to unlock them.
Key Foundational Work
Large Language Models are Zero-Shot Reasoners (2022) — contributed 3 concept elements to this direction.
Large Language Models are Zero-Shot Reasoners (2022) — contributed 1 concept elements to this direction.
Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models (2021) — contributed 1 concept elements to this direction.