Expanding the Capabilities of Reinforcement Learning via Text Feedback

Yuda Song, Lili Chen, Fahim Tajwar, Rémi Munos, Deepak Pathak, Drew Bagnell, Aarti Singh, Andrea Zanette

Reinforcement Learning (RL) has become the industry standard for aligning Large Language Models (LLMs). However, traditional RL relies on incredibly sparse signals—often just a single bit of information, like a binary "correct" or "incorrect" reward per response. If a model fails a complex mathematical proof, a simple "0" reward offers no insight into where the logic broke down. At the other end of the spectrum, "Distillation" trains models to mimic expert demonstrations, providing dense information but at a prohibitive cost.

A recent paper presented at ICML 2026 introduces Reinforcement Learning from Text Feedback (RLTF), a framework that utilizes natural language critiques to provide a "Goldilocks" level of supervision. Text feedback is richer than a simple reward score, yet far cheaper and more scalable than full human demonstrations.

The Feedback Asymmetry Problem

In real-world settings, we often have access to rich critiques during training—from users, compilers, or automated judges. However, at test time, users expect a perfect answer on the first try. Standard multi-turn RL trains a model to become a better editor (improving upon revision), but it does not necessarily make the model a smarter author on its first attempt.

RLTF bridges this gap by enabling LLMs to "internalize" natural language critiques received during training, fundamentally improving their single-turn performance at test time when no external feedback is available.

Core Methodologies

To translate training-time feedback into first-turn capability, the authors propose two novel strategies:

1. Self Distillation (RLTF-SD)

In this approach, the model generates an initial attempt, receives a critique, and produces a revised, improved second attempt. The Self Distillation objective then trains the model's first-turn policy to match its own second-turn output.

  • Innovation: To prevent "gradient collapse"—a scenario where the model stops learning because it successfully corrects itself every time—the authors implement a first-turn mean baseline. This ensures the model is consistently incentivized to improve its initial attempt until it matches the quality of the corrected version.

2. Feedback Modeling (RLTF-FM)

Here, the model is trained with an auxiliary task: predicting the exact text of the critique it is about to receive.

  • Innovation: This acts as a "representation preconditioner." By learning to predict an external judge's critique, the model develops internal features that help it avoid those mistakes in the first place. Furthermore, this mechanism enables test-time self-critique, allowing the model to generate its own internal feedback to self-correct without relying on an external judge.

Key Insights and Theoretical Grounding

The paper provides both theoretical and practical insights into why text feedback is so effective:

  • The "Low-Signal Subspace": The authors mathematically demonstrate that sparse rewards only provide learning signals in a very narrow set of representation directions. Text feedback "fills in" these missing directions, making the model's learning process geometrically "well-conditioned."
  • Variance over Bias: During distillation experiments, the researchers found that discarding "importance weighting" (a standard RL technique) actually improved training stability. In the high-dimensional space of LLM tokens, the variance of these weights becomes extreme; a slightly biased but low-variance update (akin to Advantage-Weighted Regression) yields better results.

Empirical Results

The authors evaluated RLTF across diverse domains, including the Reasoning Gym (logic puzzles), MATH500 and AIME24 (competitive math), and LitBench/WritingBench (creative writing). The proposed methods consistently outperformed standard RL baselines, specifically the GRPO algorithm:

  • Reasoning Puzzles: On the Knights and Knaves logic task, single-turn accuracy surged from 37.3% (standard RL) to 88.0% (RLTF-FM).
  • Mathematics: On the MATH500 benchmark, RLTF-FM achieved 63.6% accuracy, compared to 55.8% for the strongest RL baseline.
  • Creative Writing: RLTF-SD proved highly effective for open-ended generation, outperforming baselines on both WritingBench and LitBench.
  • Self-Correction: Models trained with Feedback Modeling successfully performed multiple rounds of self-critique at test time, yielding further performance gains entirely autonomously.

Limitations and Future Work

While RLTF presents a significant leap forward, the authors note a few limitations. The study assumes access to a high-quality "expert" feedback provider (e.g., a larger LLM); real-world human feedback, which is often noisy or subjective, may degrade performance. Additionally, the experiments focused primarily on a two-turn setup (Attempt -> Feedback -> Revision). Scaling this approach to long-form, dialogue-based training will likely require new techniques, such as feedback summarization, to manage context window limits. Finally, extending the theoretical analysis of feedback modeling to encompass the entire optimization trajectory remains an open challenge for future research.

Summary updated

Abstract

The success of RL for LLM post-training stems from an unreasonably uninformative source: a single bit of information per rollout as binary reward or preference label. At the other extreme, distillation offers dense supervision but requires demonstrations, which are costly and difficult to scale. We study natural language feedback as an intermediate signal: richer than scalar rewards, yet cheaper than complete demonstrations. Textual feedback is a natural mode of human interaction and is already abundant in many real-world settings, where users, tools, and automated judges routinely critique LLM outputs. Towards leveraging text feedback at scale, we formalize a multi-turn RL setup where text feedback is available during training but not at inference. Therefore, models must learn to internalize the feedback in order to improve their test-time single-turn performance. To do this, we propose two methods: Self Distillation, which trains the single-turn policy to match its own feedback-conditioned second-turn generations; and Feedback Modeling, which predicts the feedback as an auxiliary objective. We provide theoretical analysis on both methods, and empirically evaluate on reasoning puzzles, competition math, and creative writing tasks. Our results show that both methods consistently outperform strong baselines across benchmarks, highlighting the potential of RL with an additional source of rich supervision at scale.
BibTeX
@inproceedings{song2026expanding,
  author = {Yuda Song and Lili Chen and Fahim Tajwar and Rémi Munos and Deepak Pathak and Drew Bagnell and Aarti Singh and Andrea Zanette},
  title = {{Expanding the Capabilities of Reinforcement Learning via Text Feedback}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=3voRLZLkwU},
}
Figure