Reward Is Enough: LLMs Are In-Context Reinforcement Learners

Kefan Song, Amir Moeini, Peng Wang, Lei Gong, Rohan Chandra, Shangtong Zhang, Yanjun Qi

TL;DR

This paper reveals that reinforcement learning (RL) can spontaneously emerge during the inference phase of Large Language Models (LLMs) without any weight updates. By introducing a minimal multi-round prompting framework called ICRL prompting, the authors demonstrate that LLMs can optimize purely scalar numerical rewards to continuously improve their performance on complex reasoning and creative tasks.

What Problem Does This Paper Address?

Current approaches to test-time scaling—allowing an LLM to spend more compute at inference time to get a better answer—typically rely on language-guided self-revision (like Self-Refine or Reflexion) or complex tree-search algorithms (like Tree of Thoughts or MCTS).

However, language-guided revision often suffers from "hallucinated feedback," where the model's own text critiques introduce errors that accumulate over iterations, leading to performance collapse. Search methods, on the other hand, require heavily engineered external modules and heuristics.

The authors of this paper tackle a fundamental question: Can LLMs act as pure, in-context reinforcement learners? Specifically, can they optimize their behavior solely from numerical scalar rewards at inference time, mimicking classical RL algorithms without changing a single model weight?

Methods

The authors introduce In-Context Reinforcement Learning (ICRL) prompting, a remarkably elegant and minimal framework designed to isolate the LLM's intrinsic RL capabilities.

Unlike methods that provide verbose textual critiques, ICRL prompting only appends previous attempts formatted as state-action-reward tuples (e.g., appending Reward: [score] directly after a generated response) into the context window. To guide the model's inference-time RL process, they introduce simple meta-instructions to handle the classic exploration-exploitation trade-off:

  1. ICRL Preset: Alternates strictly between exploration prompts (asking the model to try a completely different approach) on even episodes and exploitation prompts (asking the model to generate the best response based on the highest-reward history) on odd episodes.
  2. ICRL Autonomous: Provides a combined instruction and lets the LLM decide for itself whether to explore or exploit based on the history.

Crucially, this setup completely excludes textual gradients, experience replay, or external code modules, relying entirely on the transformer's ability to implicitly run an RL algorithm during its forward pass.

Figure 1: ICRL Prompting. At each episode k+1, LLM generates action tokens based on previous experiences up to k, and receives numerical re- wards either from itself as the evaluator or from the environment. At the end of the episode, the rewards are then concatenated with the action to- kens and placed back into the context.

Datasets

The framework was evaluated across several diverse and challenging benchmarks:

  • Game of 24: A mathematical puzzle where the model must combine four given numbers using basic arithmetic to reach exactly 24.
  • Creative Writing: Crafting a coherent four-paragraph narrative that logically concludes each paragraph with a specific, randomly sampled sentence.
  • ScienceWorld: A complex, text-based interactive environment simulating elementary school science experiments.
  • Olympiad-Level Mathematics: High-difficulty competition problems from the American Invitational Mathematics Examination (AIME) and the Harvard–MIT Mathematics Tournament (HMMT).
  • Unseen Paper Abstract Generation: A custom task using 30 arXiv papers published after the model's training cutoff date to test learning completely outside its parametric memory.

Key Insights and Ideas

  • The "Duck Test" for Inference RL: The authors proved the emergence of true RL behavior by showing that performance drops significantly if rewards are omitted (set to zero) or if the context window is artificially shortened. The model isn't just randomly guessing; it is actively maximizing the scalar reward signal.
  • Self-Evaluation is Enough: Remarkably, ICRL prompting succeeds even when the scalar rewards are generated by the same LLM (prompted separately as a judge). This supports the widely held belief that evaluation is fundamentally easier than generation for language models.
  • Learning Beyond Parametric Memory: In the unseen arXiv abstract task, traditional search and self-correction methods quickly plateaued because they couldn't generate new knowledge. ICRL, however, steadily improved ROUGE scores over 200 iterations, proving it can extract and exploit information purely from an external scalar reward signal.
  • Reward-Aware Attention Heads: A mechanistic analysis of the transformer layers revealed that roughly 29.1% of attention heads dynamically correlate with reward signals. Some heads specifically track successful, high-reward tokens, while others focus entirely on analyzing failures.

Key Results

  • Game of 24: ICRL Preset achieved a stunning 90% success rate, vastly outperforming Best-of-N (49%), Self-Refine (47%), and Reflexion (44%).
  • Creative Writing: ICRL achieved a length-controlled win rate of 59.48% against Reflexion and 86.32% against Self-Refine.
  • ScienceWorld: ICRL prompting outperformed baseline methods by approximately 20% after sufficient iterations.
  • Olympiad Math: ICRL delivered substantial performance boosts (up to 10–20 points over the base model) while using three orders of magnitude less compute for feedback compared to verbal self-correction methods.

Limitations and Future Work

  • Context Window Constraints: Because ICRL relies on appending all previous attempts and rewards into the prompt, it is fundamentally bounded by the model's maximum context length. Long-horizon tasks will eventually saturate the window.
  • Future Directions: The authors suggest exploring training-time interventions—such as specific algorithmic distillation or pre-training on RL trajectories—to explicitly enhance and unlock this latent in-context RL capability in future open-source base models.

Summary updated

Abstract

Reinforcement learning (RL) is a framework for solving sequential decision-making problems. In this work, we demonstrate that, surprisingly, RL emerges during the inference time of large language models (LLMs), a phenomenon we term in-context RL (ICRL). To reveal this capability, we introduce a simple multi-round prompting framework, we call ICRL prompting, for inference-time self-improvement. The goal of ICRL prompting is to guide LLMs to perform reinforcement learning during inference for self-improvement on a given task. After each response, the model receives numerical scalar feedback, denoted as a reward. In the next round, we prompt the LLM again together with a context that concatenates all prior responses and their associated rewards. We consistently observe that response quality improves as the context grows. In other words, the LLM can optimize scalar reward signals during inference, exhibiting behavior analogous to reinforcement learning. We evaluate ICRL prompting on Game of 24, creative writing, ScienceWorld, and Olympiad-level math competitions (AIME and HMMT), demonstrating significant improvements over baselines such as Self-Refine and Reflexion. Notably, even when the reward signals are generated by the same LLM, ICRL prompting still improves performance, highlighting a promising new paradigm for test-time scaling.
BibTeX
@inproceedings{song2026reward,
  author = {Kefan Song and Amir Moeini and Peng Wang and Lei Gong and Rohan Chandra and Shangtong Zhang and Yanjun Qi},
  title = {{Reward Is Enough: LLMs Are In-Context Reinforcement Learners}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=keCXNHOe4W},
}
Figure