TL;DR
BRAT is a new reinforcement learning objective that stabilizes "multi-pass" training—where the same batch of generated responses is used for multiple gradient updates—by preventing the model from overfitting to small rollout groups. By combining a smooth, bounded token-level objective with a parameter-free reward normalization, BRAT avoids training collapse and achieves higher accuracy with lower variance on complex mathematical reasoning benchmarks.
What Problem Does This Paper Address?
Reinforcement Learning with Verifiable Rewards (RLVR) has become the go-to method for teaching Large Language Models (LLMs) how to reason through math and code. However, generating responses (rollouts) from an LLM is incredibly slow and expensive. To save time, researchers often reuse the same batch of rollouts for several gradient updates (multi-pass training).
The problem is that this reuse is notoriously fragile. Existing methods like PPO or GRPO often suffer from "training collapse" or "policy drift," where the model's performance suddenly tanks as it moves too far away from the policy that originally generated the samples. Until now, we didn't fully understand why this happens. This paper identifies the culprit: extreme overfitting. The authors found that the update direction calculated from a small group of samples is nearly an order of magnitude less effective on new, unseen samples than on the training samples themselves.
Methods
The authors propose Bounded Residual-Aligned Training (BRAT), which introduces two key innovations to handle multi-pass training:
- Symmetric-Maximum (SymMax) Group Normalization: Standard methods like GRPO normalize rewards by dividing by the group's standard deviation. This can blow up the importance of rare outcomes (e.g., one correct answer in a group of failures). BRAT instead divides by the maximum magnitude of the centered rewards. This keeps the weights bounded between $[-1, 1]$ and ensures they sum to zero without needing extra hyperparameters.
- Directional Saturating (DS) Token Objective: Instead of the "hard clipping" used in PPO (which cuts off gradients abruptly) or unclipped updates (which lead to collapse), BRAT uses a smooth gradient multiplier. As the model's probability for a token moves toward a "trust boundary," the gradient weight decays linearly to zero. This prevents any single token from being pushed too far, effectively "saturating" the update.
Datasets
The models were post-trained using verified problems from the NuminaMath-1.5 dataset. Evaluation was conducted across six rigorous mathematical reasoning benchmarks totaling 1,546 problems:
- MATH-500
- OlympiadBench
- Minerva-Math
- AMC 2023
- AIME 2024
- AIME 2025
Key Insights and Ideas
- The Alignment Gap: The most striking discovery is the "In-Sample vs. Held-Out Alignment Gap." The authors measured "directional efficiency"—how well a policy update actually moves the model toward higher rewards. They found that efficiency drops by ~9x when moving from the rollouts used for the update to new rollouts of the same prompts. This proves that multi-pass training is essentially "fitting the noise" of a specific sample group.
- Smoothness over Clipping: Hard clipping (like in PPO) creates "dead zones" where the model stops learning abruptly. BRAT’s smooth decay allows for more stable, continuous progress.
- Parameter-Free Stability: Unlike many RL methods that require careful tuning of KL-penalty coefficients or clipping ranges, BRAT’s SymMax normalization provides stability "out of the box" by naturally bounding the influence of any single response.
Key Results
In head-to-head comparisons using a Qwen3-8B-Base model with 4-pass reuse:
- Superior Accuracy: BRAT achieved the highest mean accuracy on 5 out of 6 benchmarks. For example, on MATH-500, it reached 79.60%, outperforming GRPO (77.93%) and AsymRE (78.60%).
- Rock-Solid Stability: BRAT significantly reduced variance across different random seeds. On the AMC 2023 benchmark, BRAT’s standard deviation was only 1.44 pp, compared to a massive 7.50 pp for AsymRE.
- No Collapse: While unclipped methods and even some clipped variants collapsed during longer training runs (16 rounds), BRAT’s performance improved monotonically.
Limitations and Future Work
While the results are promising, the authors note a few areas for further exploration:
- Scale: The online training experiments were limited to a single base model size (8B) using LoRA adapters. Testing on larger models (70B+) and with full-parameter tuning is a logical next step.
- Theory-to-Practice Gap: The theoretical guarantees for BRAT are derived at the sequence level, but LLMs are trained at the token level. While the token-level surrogate works well in practice, a tighter theoretical bridge could be beneficial.
- Hybrid Schedules: The authors suggest that future work could combine BRAT’s stability with late-stage "greedy" refinement to squeeze out even more performance.
Summary updated