Reinforcement Learning (RL) has become the industry standard for aligning Large Language Models (LLMs). However, traditional RL relies on incredibly sparse signals—often just a single bit of information, like a binary "correct" or "incorrect" reward per response. If a model fails a complex mathematical proof, a simple "0" reward offers no insight into where the logic broke down. At the other end of the spectrum, "Distillation" trains models to mimic expert demonstrations, providing dense information but at a prohibitive cost.
A recent paper presented at ICML 2026 introduces Reinforcement Learning from Text Feedback (RLTF), a framework that utilizes natural language critiques to provide a "Goldilocks" level of supervision. Text feedback is richer than a simple reward score, yet far cheaper and more scalable than full human demonstrations.
The Feedback Asymmetry Problem
In real-world settings, we often have access to rich critiques during training—from users, compilers, or automated judges. However, at test time, users expect a perfect answer on the first try. Standard multi-turn RL trains a model to become a better editor (improving upon revision), but it does not necessarily make the model a smarter author on its first attempt.
RLTF bridges this gap by enabling LLMs to "internalize" natural language critiques received during training, fundamentally improving their single-turn performance at test time when no external feedback is available.
Core Methodologies
To translate training-time feedback into first-turn capability, the authors propose two novel strategies:
1. Self Distillation (RLTF-SD)
In this approach, the model generates an initial attempt, receives a critique, and produces a revised, improved second attempt. The Self Distillation objective then trains the model's first-turn policy to match its own second-turn output.
- Innovation: To prevent "gradient collapse"—a scenario where the model stops learning because it successfully corrects itself every time—the authors implement a first-turn mean baseline. This ensures the model is consistently incentivized to improve its initial attempt until it matches the quality of the corrected version.
2. Feedback Modeling (RLTF-FM)
Here, the model is trained with an auxiliary task: predicting the exact text of the critique it is about to receive.
- Innovation: This acts as a "representation preconditioner." By learning to predict an external judge's critique, the model develops internal features that help it avoid those mistakes in the first place. Furthermore, this mechanism enables test-time self-critique, allowing the model to generate its own internal feedback to self-correct without relying on an external judge.
Key Insights and Theoretical Grounding
The paper provides both theoretical and practical insights into why text feedback is so effective:
- The "Low-Signal Subspace": The authors mathematically demonstrate that sparse rewards only provide learning signals in a very narrow set of representation directions. Text feedback "fills in" these missing directions, making the model's learning process geometrically "well-conditioned."
- Variance over Bias: During distillation experiments, the researchers found that discarding "importance weighting" (a standard RL technique) actually improved training stability. In the high-dimensional space of LLM tokens, the variance of these weights becomes extreme; a slightly biased but low-variance update (akin to Advantage-Weighted Regression) yields better results.
Empirical Results
The authors evaluated RLTF across diverse domains, including the Reasoning Gym (logic puzzles), MATH500 and AIME24 (competitive math), and LitBench/WritingBench (creative writing). The proposed methods consistently outperformed standard RL baselines, specifically the GRPO algorithm:
- Reasoning Puzzles: On the Knights and Knaves logic task, single-turn accuracy surged from 37.3% (standard RL) to 88.0% (RLTF-FM).
- Mathematics: On the MATH500 benchmark, RLTF-FM achieved 63.6% accuracy, compared to 55.8% for the strongest RL baseline.
- Creative Writing: RLTF-SD proved highly effective for open-ended generation, outperforming baselines on both WritingBench and LitBench.
- Self-Correction: Models trained with Feedback Modeling successfully performed multiple rounds of self-critique at test time, yielding further performance gains entirely autonomously.
Limitations and Future Work
While RLTF presents a significant leap forward, the authors note a few limitations. The study assumes access to a high-quality "expert" feedback provider (e.g., a larger LLM); real-world human feedback, which is often noisy or subjective, may degrade performance. Additionally, the experiments focused primarily on a two-turn setup (Attempt -> Feedback -> Revision). Scaling this approach to long-form, dialogue-based training will likely require new techniques, such as feedback summarization, to manage context window limits. Finally, extending the theoretical analysis of feedback modeling to encompass the entire optimization trajectory remains an open challenge for future research.
Summary updated