GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models

Peizheng Guo, Jianqi Zhang, Xingyu Zhang, Yun Fan, Jiahuan Zhou, Changwen Zheng, Wenwen QiangarXiv 2026 · arXiv

TL;DR

Group Relative Policy Optimization (GRPO) is a popular method for post-training reasoning LLMs, but it suffers when gradients from different queries in a mini-batch conflict. This paper introduces Gradient Uncertainty-aware Policy Optimization (GUPO), a Bayesian framework that models group gradients as probability distributions and uses Dirichlet-based uncertainty to downweight unreliable, conflicting updates. GUPO consistently improves performance across multiple mathematical reasoning benchmarks.

What Problem Does This Paper Address?

Post-training Large Language Models (LLMs) for complex reasoning often relies on Reinforcement Learning methods like GRPO. In GRPO, a mini-batch contains multiple queries, and for each query, a group of responses is sampled to compute a "group gradient." Standard GRPO simply averages these group gradients to update the model.

However, the authors discover a major flaw: group gradients from different queries within the same mini-batch frequently point in conflicting directions (i.e., they have negative cosine similarity). Empirically, mini-batches with high conflict rates lead to significantly less effective policy updates or even negative validation gains. Because standard GRPO treats these gradients as deterministic and equally reliable, highly erratic or uncertain gradients can disproportionately skew the optimization path.

Empirical evidence of group-gradient conflicts in GRPO and their negative impact on validation performance.

Methods

To solve this, the authors propose GUPO (Gradient Uncertainty-aware Policy Optimization). Instead of treating group gradients as fixed, deterministic vectors, GUPO treats them as random variables with underlying probability distributions. The pipeline consists of three main steps:

  1. Posterior Parameter Estimation: Because computing a full Hessian matrix for an LLM is computationally impossible, the authors use a diagonal empirical Fisher information matrix to construct a tractable Gaussian approximation of the policy parameter posterior distribution.
  2. Group Gradient Distribution Estimation: Using Monte Carlo sampling, they draw parameter sets from the estimated posterior and compute the corresponding group gradients. This allows them to estimate the mean and element-wise variance for each query's gradient.
  3. Uncertainty-Aware Aggregation: Using a Dirichlet-based evidential formulation inspired by subjective logic, the element-wise variance is converted into a single group-level uncertainty score ($u_b$). When aggregating the mini-batch gradient, GUPO dynamically adjusts the weights to favor low-uncertainty (highly reliable) gradients while attenuating the influence of highly uncertain ones.
Overview of the GUPO framework, spanning rollout, Bayesian gradient distribution estimation, and uncertainty-aware policy optimization.

Datasets

The authors evaluated GUPO across six prominent mathematical reasoning benchmarks:

  • GSM8K: Grade-school math word problems.
  • MATH500 & MinervaMATH: Diverse, complex mathematical problem-solving tasks.
  • AMC 2023, AIME 2024, & AIME 2025: High-level, competition-grade mathematical reasoning datasets.

No new datasets were introduced; the focus was entirely on evaluating post-training performance on established reasoning baselines.

Key Insights and Ideas

  • Gradient Conflict is Pervasive: In standard GRPO mini-batches, nearly 46.5% of all group-gradient pairs exhibit negative cosine similarity (directional conflict).
  • Reliability $\neq$ Correctness: The paper reframes the problem from "which gradient is right?" to "which gradient is reliable?". Reliability is measured by how consistently a gradient direction is maintained across parameter perturbations. If the distribution is tightly concentrated, it's reliable.
  • Variance as a Calibration Tool: By mapping high gradient variance to high uncertainty, the model can automatically protect itself from being derailed by anomalous or highly erratic query groups within a mini-batch.

Key Results

GUPO was tested against standard GRPO and several recent advanced variants (like GVPO, Dr.GRPO, GCPO) using three base models: DeepScaleR-1.5B-Preview, DeepSeek-R1-Distill-Qwen-1.5B, and DeepSeek-R1-Distill-Qwen-7B.

  • Consistent Performance Boost: GUPO achieved the highest average score across all benchmarks for all three base models. For instance, on DeepScaleR-1.5B-Preview, GUPO reached an average score of 63.4%, outperforming standard GRPO (60.7%) and the next best baseline, GCPO (62.3%).
  • Mitigating High-Conflict Batches: Validation analysis showed that GUPO significantly improves validation Negative Log-Likelihood (NLL) gains specifically during high-conflict mini-batches, proving that the uncertainty calibration successfully rescues otherwise destructive updates.
Validation performance gains of GUPO over GRPO, demonstrating superior stability in high-conflict scenarios.

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs) for reasoning. In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update. However, these group gradients can point in conflicting directions. Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates, motivating the need for a reliable aggregated update direction under such conflicts. Standard GRPO aggregation treats the realized group gradients as deterministic contributions and does not account for differences in their reliability during aggregation. To address this issue, we propose Gradient Uncertainty-Aware Policy Optimization (GUPO), which models each group gradient as a random variable under a Bayesian formulation and estimates its probability distribution. GUPO then derives gradient uncertainty using a Dirichlet-based formulation and uses it to calibrate the contribution of each group gradient during aggregation. Extensive experiments on multiple benchmarks demonstrate the effectiveness of GUPO.
BibTeX
@misc{guo2026gupo,
  author = {Peizheng Guo and Jianqi Zhang and Xingyu Zhang and Yun Fan and Jiahuan Zhou and Changwen Zheng and Wenwen Qiang},
  title = {{GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models}},
  year = {2026},
  eprint = {2608.17411},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.17411},
}
Figure