TL;DR
Post-training large language models by combining supervised fine-tuning (SFT) and reinforcement learning (RL) typically relies on rigid, hand-tuned mixing schedules. This paper introduces the Guided Adaptive Controller (GAC), a noise-aware framework that dynamically adjusts the SFT-RL mixing weight based on real-time gradient uncertainty and objective disagreement. By optimizing this balance online with negligible overhead, GAC stabilizes training, prevents reward hacking, and significantly boosts performance across math, code, and reasoning tasks.
What Problem Does This Paper Address?
Aligning Large Language Models (LLMs) usually involves a hybrid post-training approach: Supervised Fine-Tuning (SFT) stabilizes formatting and retains expert behavior, while Reinforcement Learning (RL) maximizes reward-seeking and reasoning capabilities. However, SFT and RL objectives often conflict, and their relative gradient noise levels shift dynamically throughout the training process.
Current methods rely on fixed or heuristic mixing schedules. If the mixing weight doesn't adapt when RL exploration becomes highly volatile, the model can suffer from "entropy collapse" or fall prey to reward hacking—such as generating excessively long responses just to exploit length biases. The authors tackle the challenge of designing a principled, dynamic controller that can sense gradient noise and disagreement on the fly, adjusting the training mixture smoothly without incurring heavy computational costs.
Methods
The core of GAC is a closed-form optimal mixing weight ($\mu^*$) derived by minimizing a Mean Squared Error (MSE) upper bound between the mixed stochastic gradient and an idealized target gradient. The resulting formula elegantly balances SFT gradient noise variance ($\sigma_s^2$), RL gradient noise variance ($\sigma_r^2$), and the spatial disagreement between the SFT and RL gradients ($\Delta g^2$).
Because computing exact full-gradient statistics at every step is prohibitively expensive, GAC introduces computationally lightweight proxy signals calculated directly from existing training tensors:
- RL Uncertainty ($\sigma_r^2$): Measured via sequence-level advantage dispersion within the mini-batch.
- SFT Uncertainty ($\sigma_s^2$): Measured using length-normalized, trimmed negative log-likelihood (NLL) variance.
- Gradient Disagreement ($\Delta \tilde{g}^2$): Estimated using z-score normalized token-level coefficient differences.
To prevent erratic updates from noisy proxies, the raw estimator is wrapped in a Guided Adaptive Controller consisting of three guardrails: Exponential Moving Average (EMA) smoothing, blending with a standard schedule prior (warmup + cosine), and a strict per-step change cap. GAC also integrates token-wise SFT reweighting adopted from prior work (CHORD).
Datasets
The authors sampled training data from OpenR1-Math-220k (5,000 SFT instances and 20,000 RL prompts).
To evaluate cross-domain generalization, they tested the models on a robust suite of benchmarks:
- Mathematics: AMC, AIME24, AIME25
- Code Generation: MBPP, HumanEval
- Scientific & Logical Reasoning: MMLU-Pro, GPQA, SciBench, and logical subsets of Big-Bench Hard (BBH).
Key Insights and Ideas
- Cheap Proxies Mirror True Gradients: One of the paper's most practical findings is that simple mini-batch statistics (like advantage variance) correlate remarkably well ($r \approx 0.8$) with actual, expensive full-gradient noise properties.
- Noise-Tracking Beats Pure KL-Tracking: Rather than simply reacting to KL divergence from a reference model, GAC directly tracks RL noise spikes. When RL exploration enters a volatile phase, GAC automatically shifts weight back toward SFT, anchoring the policy and preventing catastrophic optimization failures.
- Scale Amplifies the Benefits: As models grow from 1.5B to 14B parameters, the dynamic range of SFT/RL noise widens. Consequently, the performance gains of GAC over fixed schedules actually increase with model size, making it highly relevant for scaling post-training pipelines.
Key Results
GAC demonstrates superior performance and training stability compared to strong rule-based and multi-objective baselines:
- Performance Boosts: On a Qwen2.5-7B-Instruct base, GAC combined with token reweighting outperformed the best rule-based controller by +3.0 percentage points (pp) on AMC and surpassed the strong hybrid baseline HPT by +3.8pp across math, code, science, and logic tasks.
- Mitigating Reward Hacking: As shown in the training dynamics, GAC successfully suppresses the dramatic response-length spikes and policy-gradient oscillations that plague fixed-schedule baselines like QCM and HPCD.
- Enhanced Stability: GAC reduced the total KL-drift area by 28% and slashed large, destabilizing weight-shift events ($|\Delta \mu| > 0.02$) by 73% relative to constant mixing.
- Negligible Overhead: Because it reuses existing tensors, GAC adds fewer than 3 EMA scalars, resulting in <1% wall-time overhead and zero measurable memory increase.
Limitations and Future Work
While theoretically grounded, the stability analysis of GAC relies on idealized assumptions (such as $L$-smooth losses and unbiased proxies) that do not perfectly hold in complex, real-world LLM training environments. Additionally, the framework was validated exclusively on tasks with verifiable rewards (e.g., mathematics and coding outcomes where correctness can be programmatically checked). The authors note that extending GAC to open-ended alignment tasks—which rely on noisier, learned human preference reward models—remains an open and critical direction for future research.
Summary updated