BRAT: Bounded Residual-Aligned Training for Stable Multi-Pass LLM Post-Training

Xunlan Zhou, Yuzhe Zhao, Xuhong Huang, Feng Chen, Xuanlin Chen, Lang Feng, Bo AnarXiv 2026 · arXiv

TL;DR

BRAT is a new reinforcement learning objective that stabilizes "multi-pass" training—where the same batch of generated responses is used for multiple gradient updates—by preventing the model from overfitting to small rollout groups. By combining a smooth, bounded token-level objective with a parameter-free reward normalization, BRAT avoids training collapse and achieves higher accuracy with lower variance on complex mathematical reasoning benchmarks.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR) has become the go-to method for teaching Large Language Models (LLMs) how to reason through math and code. However, generating responses (rollouts) from an LLM is incredibly slow and expensive. To save time, researchers often reuse the same batch of rollouts for several gradient updates (multi-pass training).

The problem is that this reuse is notoriously fragile. Existing methods like PPO or GRPO often suffer from "training collapse" or "policy drift," where the model's performance suddenly tanks as it moves too far away from the policy that originally generated the samples. Until now, we didn't fully understand why this happens. This paper identifies the culprit: extreme overfitting. The authors found that the update direction calculated from a small group of samples is nearly an order of magnitude less effective on new, unseen samples than on the training samples themselves.

Methods

The authors propose Bounded Residual-Aligned Training (BRAT), which introduces two key innovations to handle multi-pass training:

  1. Symmetric-Maximum (SymMax) Group Normalization: Standard methods like GRPO normalize rewards by dividing by the group's standard deviation. This can blow up the importance of rare outcomes (e.g., one correct answer in a group of failures). BRAT instead divides by the maximum magnitude of the centered rewards. This keeps the weights bounded between $[-1, 1]$ and ensures they sum to zero without needing extra hyperparameters.
  2. Directional Saturating (DS) Token Objective: Instead of the "hard clipping" used in PPO (which cuts off gradients abruptly) or unclipped updates (which lead to collapse), BRAT uses a smooth gradient multiplier. As the model's probability for a token moves toward a "trust boundary," the gradient weight decays linearly to zero. This prevents any single token from being pushed too far, effectively "saturating" the update.
The BRAT token gradient multiplier (left) decays linearly to zero at the trust boundary, unlike the discontinuous clipping of GRPO or the constant weight of AsymRE.

Datasets

The models were post-trained using verified problems from the NuminaMath-1.5 dataset. Evaluation was conducted across six rigorous mathematical reasoning benchmarks totaling 1,546 problems:

  • MATH-500
  • OlympiadBench
  • Minerva-Math
  • AMC 2023
  • AIME 2024
  • AIME 2025

Key Insights and Ideas

  • The Alignment Gap: The most striking discovery is the "In-Sample vs. Held-Out Alignment Gap." The authors measured "directional efficiency"—how well a policy update actually moves the model toward higher rewards. They found that efficiency drops by ~9x when moving from the rollouts used for the update to new rollouts of the same prompts. This proves that multi-pass training is essentially "fitting the noise" of a specific sample group.
Figure 1: In-sample vs. held-out residual alignment diagnostic across three base models (128 prompts, η = 1.0). (a) First-order preconditioned gain Adiag = ⟨ˆgη,16, ˆd⟩on the K = 16
  • Smoothness over Clipping: Hard clipping (like in PPO) creates "dead zones" where the model stops learning abruptly. BRAT’s smooth decay allows for more stable, continuous progress.
  • Parameter-Free Stability: Unlike many RL methods that require careful tuning of KL-penalty coefficients or clipping ranges, BRAT’s SymMax normalization provides stability "out of the box" by naturally bounding the influence of any single response.

Key Results

In head-to-head comparisons using a Qwen3-8B-Base model with 4-pass reuse:

  • Superior Accuracy: BRAT achieved the highest mean accuracy on 5 out of 6 benchmarks. For example, on MATH-500, it reached 79.60%, outperforming GRPO (77.93%) and AsymRE (78.60%).
  • Rock-Solid Stability: BRAT significantly reduced variance across different random seeds. On the AMC 2023 benchmark, BRAT’s standard deviation was only 1.44 pp, compared to a massive 7.50 pp for AsymRE.
  • No Collapse: While unclipped methods and even some clipped variants collapsed during longer training runs (16 rounds), BRAT’s performance improved monotonically.
Figure 3: BRAT (yellow) consistently achieves higher gains over the base model with much lower cross-seed standard deviation than competitors.

Limitations and Future Work

While the results are promising, the authors note a few areas for further exploration:

  • Scale: The online training experiments were limited to a single base model size (8B) using LoRA adapters. Testing on larger models (70B+) and with full-parameter tuning is a logical next step.
  • Theory-to-Practice Gap: The theoretical guarantees for BRAT are derived at the sequence level, but LLMs are trained at the token level. While the token-level surrogate works well in practice, a tighter theoretical bridge could be beneficial.
  • Hybrid Schedules: The authors suggest that future work could combine BRAT’s stability with late-stage "greedy" refinement to squeeze out even more performance.

Summary updated

Abstract

Reinforcement learning for large language model post-training often reuses each batch of sampled rollouts for several gradient passes to amortize the cost of generation. Under such multi-pass reuse, existing objectives fail in different ways: unclipped updates collapse as the policy drifts away from the sampling policy, adding a negative baseline avoids collapse but reinforces every sampled response, and PPO-style ratio clipping cuts off gradients abruptly. We first ask why reuse is fragile. Measuring how well an update estimated from a small rollout group aligns with the KL-regularized target on held-out rollouts of the same prompts, we find that this alignment drops by nearly an order of magnitude from in-sample to held-out rollouts across three base models, suggesting that repeated passes largely overfit the sampled group. Motivated by this finding, we propose Bounded Residual-Aligned Training (BRAT), motivated by a proximal update toward a bounded reweighting of the sampling policy. BRAT assigns each token a gradient weight that decreases linearly with its probability ratio and vanishes at a trust-region boundary, and normalizes group-centered rewards by their maximum magnitude, keeping advantages zero-mean and bounded without extra hyperparameters. With four reuse passes, BRAT avoids training collapse, limits policy drift, and achieves higher average accuracy with lower seed variance on most benchmarks.
BibTeX
@misc{zhou2026brat,
  author = {Xunlan Zhou and Yuzhe Zhao and Xuhong Huang and Feng Chen and Xuanlin Chen and Lang Feng and Bo An},
  title = {{BRAT: Bounded Residual-Aligned Training for Stable Multi-Pass LLM Post-Training}},
  year = {2026},
  eprint = {2611.10873},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10873},
}
Figure