OBLR-PO: A Theoretical Framework for Stable Reinforcement Learning

Zixun Huang, Jiayi Sheng, Zeyu ZhengarXiv 2025 · DOI · arXiv

Introduction

Reinforcement learning from human feedback (RLHF) has become the cornerstone of aligning large language models (LLMs) with human intent and reasoning capabilities. However, the practical application of RL algorithms like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) often feels more like an art than a science. Practitioners frequently rely on heuristic "tricks"—such as manual reward scaling or arbitrary learning rate schedules—to prevent training collapse.

OBLR-PO (Optimal Baseline and Learning-Rate Policy Optimization) addresses this by moving RL for LLMs away from "vibes-based" heuristics toward a rigorous theoretical foundation. By deriving an optimization-loss upper bound, the authors introduce an adaptive learning rate governed by the Signal-to-Noise Ratio (SNR) and a novel "gradient-weighted" baseline that significantly improves training stability and performance on complex reasoning tasks.

Theoretical Foundations

The OBLR-PO framework is built upon three core theoretical pillars designed to replace empirical guesswork with mathematical guarantees.

1. Unified Statistical Analysis

The authors provide a unified framework that characterizes the bias and variance of common policy gradient estimators. By analyzing algorithms like PPO, RLOO, and GRPO under mild assumptions, they identify the statistical properties that lead to instability.

2. SNR-Governed Learning Rates

A critical innovation in OBLR-PO is the use of an adaptive learning rate schedule governed by the Signal-to-Noise Ratio (SNR) of the gradients. In traditional RLHF, learning rates are often fixed or follow a simple linear decay. OBLR-PO instead treats the learning rate as a "trust factor." If the "signal" (the true gradient) is strong relative to the "noise" (variance), the algorithm takes a larger step. Conversely, if noise dominates, the step size is reduced to prevent catastrophic updates.

3. Optimal Gradient-Weighted Baseline

Variance reduction is typically achieved using baselines, which are often simple averages of rewards (as seen in GRPO). The authors mathematically prove that a simple average is suboptimal. They propose a gradient-weighted estimator as the mathematically optimal baseline. This approach assigns higher weights to samples that provide richer information about the model's parameter updates, leading to more efficient variance reduction.

Experimental Evaluation

The researchers evaluated OBLR-PO using the Qwen3-4B-Base and Qwen3-8B-Base models. The evaluation focused on high-difficulty mathematical reasoning benchmarks, which are particularly sensitive to training stability:

  • GSM8K & MATH500: Standard and competition-level math word problems.
  • OlympiadBench: Multimodal scientific problems at the Olympiad level.
  • AIME25 & AMC23: Advanced mathematics competition problems from 2023 and 2025.

Results and Stability Analysis

OBLR-PO consistently outperformed existing methods (PPO, GRPO, ReMax, RLOO) in both final accuracy and training stability.

Performance Gains

On the Qwen3-8B-Base model, OBLR-PO achieved a remarkable 90.8% on GSM8K and 70.4% on MATH500. These results represent a significant improvement over standard GRPO, demonstrating that theoretical optimization translates directly into reasoning performance.

Comparison of training advantage across Qwen3-4B and Qwen3-8B models, demonstrating the improved stability and higher reward signals of OBLR-PO compared to GRPO.

Measurable Stability

One of the most striking findings is the impact on training dynamics. While GRPO often exhibits volatile "spikes" in gradient norms and loss, OBLR-PO maintains a steady, downward trajectory. This stability is not merely qualitative; it is reflected in the significantly lower gradient norms observed throughout the training process.

Gradient norm trajectories during training. OBLR-PO maintains significantly lower and more stable gradient norms, indicating a more controlled optimization process.

The adaptive learning rate allowed the model to converge more reliably without the need for extensive, model-specific hyperparameter tuning, making the training process more efficient and scalable.

Conclusion and Future Work

While OBLR-PO marks a significant advancement, the authors identify areas for further refinement. The current proofs rely on the assumption of L-smoothness in the log-likelihood function, which may not perfectly describe billion-parameter transformers. Additionally, the framework does not yet explicitly account for KL divergence constraints, which are vital for maintaining model identity during RLHF. Future work will focus on integrating these dynamics into the theory and closing the gap between the optimized upper bound and the actual loss landscape.

Summary updated

BibTeX
@misc{huang2025oblr,
  author = {Zixun Huang and Jiayi Sheng and Zeyu Zheng},
  title = {{OBLR-PO: A Theoretical Framework for Stable Reinforcement Learning}},
  year = {2025},
  eprint = {2511.23310},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.23310},
}
Figure