TL;DR
This paper reveals that reinforcement learning (RL) can spontaneously emerge during the inference phase of Large Language Models (LLMs) without any weight updates. By introducing a minimal multi-round prompting framework called ICRL prompting, the authors demonstrate that LLMs can optimize purely scalar numerical rewards to continuously improve their performance on complex reasoning and creative tasks.
What Problem Does This Paper Address?
Current approaches to test-time scaling—allowing an LLM to spend more compute at inference time to get a better answer—typically rely on language-guided self-revision (like Self-Refine or Reflexion) or complex tree-search algorithms (like Tree of Thoughts or MCTS).
However, language-guided revision often suffers from "hallucinated feedback," where the model's own text critiques introduce errors that accumulate over iterations, leading to performance collapse. Search methods, on the other hand, require heavily engineered external modules and heuristics.
The authors of this paper tackle a fundamental question: Can LLMs act as pure, in-context reinforcement learners? Specifically, can they optimize their behavior solely from numerical scalar rewards at inference time, mimicking classical RL algorithms without changing a single model weight?
Methods
The authors introduce In-Context Reinforcement Learning (ICRL) prompting, a remarkably elegant and minimal framework designed to isolate the LLM's intrinsic RL capabilities.
Unlike methods that provide verbose textual critiques, ICRL prompting only appends previous attempts formatted as state-action-reward tuples (e.g., appending Reward: [score] directly after a generated response) into the context window. To guide the model's inference-time RL process, they introduce simple meta-instructions to handle the classic exploration-exploitation trade-off:
- ICRL Preset: Alternates strictly between exploration prompts (asking the model to try a completely different approach) on even episodes and exploitation prompts (asking the model to generate the best response based on the highest-reward history) on odd episodes.
- ICRL Autonomous: Provides a combined instruction and lets the LLM decide for itself whether to explore or exploit based on the history.
Crucially, this setup completely excludes textual gradients, experience replay, or external code modules, relying entirely on the transformer's ability to implicitly run an RL algorithm during its forward pass.
Datasets
The framework was evaluated across several diverse and challenging benchmarks:
- Game of 24: A mathematical puzzle where the model must combine four given numbers using basic arithmetic to reach exactly 24.
- Creative Writing: Crafting a coherent four-paragraph narrative that logically concludes each paragraph with a specific, randomly sampled sentence.
- ScienceWorld: A complex, text-based interactive environment simulating elementary school science experiments.
- Olympiad-Level Mathematics: High-difficulty competition problems from the American Invitational Mathematics Examination (AIME) and the Harvard–MIT Mathematics Tournament (HMMT).
- Unseen Paper Abstract Generation: A custom task using 30 arXiv papers published after the model's training cutoff date to test learning completely outside its parametric memory.
Key Insights and Ideas
- The "Duck Test" for Inference RL: The authors proved the emergence of true RL behavior by showing that performance drops significantly if rewards are omitted (set to zero) or if the context window is artificially shortened. The model isn't just randomly guessing; it is actively maximizing the scalar reward signal.
- Self-Evaluation is Enough: Remarkably, ICRL prompting succeeds even when the scalar rewards are generated by the same LLM (prompted separately as a judge). This supports the widely held belief that evaluation is fundamentally easier than generation for language models.
- Learning Beyond Parametric Memory: In the unseen arXiv abstract task, traditional search and self-correction methods quickly plateaued because they couldn't generate new knowledge. ICRL, however, steadily improved ROUGE scores over 200 iterations, proving it can extract and exploit information purely from an external scalar reward signal.
- Reward-Aware Attention Heads: A mechanistic analysis of the transformer layers revealed that roughly 29.1% of attention heads dynamically correlate with reward signals. Some heads specifically track successful, high-reward tokens, while others focus entirely on analyzing failures.
Key Results
- Game of 24: ICRL Preset achieved a stunning 90% success rate, vastly outperforming Best-of-N (49%), Self-Refine (47%), and Reflexion (44%).
- Creative Writing: ICRL achieved a length-controlled win rate of 59.48% against Reflexion and 86.32% against Self-Refine.
- ScienceWorld: ICRL prompting outperformed baseline methods by approximately 20% after sufficient iterations.
- Olympiad Math: ICRL delivered substantial performance boosts (up to 10–20 points over the base model) while using three orders of magnitude less compute for feedback compared to verbal self-correction methods.
Limitations and Future Work
- Context Window Constraints: Because ICRL relies on appending all previous attempts and rewards into the prompt, it is fundamentally bounded by the model's maximum context length. Long-horizon tasks will eventually saturate the window.
- Future Directions: The authors suggest exploring training-time interventions—such as specific algorithmic distillation or pre-training on RL trajectories—to explicitly enhance and unlock this latent in-context RL capability in future open-source base models.
Summary updated