SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization

Minghan Chen, Guikun Chen, Wenguan Wang, Yi Yang

Abstract

Reinforcement Learning (RL) has become a cornerstone for enhancing the reasoning capabilities of Large Language Models (LLMs). However, standard algorithms like Group Relative Policy Optimization (GRPO) treat all training prompts with equal weight, ignoring the inherent uncertainty in model responses. We introduce SEED-GRPO (Semantic Entropy EnhanceD GRPO), a novel framework that integrates semantic entropy to modulate policy updates. By scaling down updates for questions where the model exhibits high inconsistency, SEED-GRPO improves training stability and reasoning performance. Our results show that a 7B parameter model trained with SEED-GRPO can outperform much larger 32B models on challenging mathematical benchmarks like AIME24.

Introduction

The recent success of models like DeepSeek-R1 and OpenAI’s o1 highlights the power of Reinforcement Learning in boosting logical reasoning. A popular approach is Group Relative Policy Optimization (GRPO), which evaluates multiple sampled answers for a single prompt to calculate relative advantages. Despite its efficiency, vanilla GRPO suffers from a "blind spot": it does not distinguish between a model that is confident and one that is merely lucky.

When an LLM provides wildly different answers to the same problem, it signals high uncertainty. Standard GRPO ignores this signal, potentially leading the model to overfit to noisy rewards or "lucky" correct guesses. This lack of nuance can result in unstable training trajectories and suboptimal reasoning capabilities.

Comparison of certain and uncertain model responses. In certain cases, the model consistently reaches the correct answer. In uncertain cases, responses are semantically diverse and often incorrect.

Methodology: Uncertainty-Aware Training

SEED-GRPO addresses these limitations by introducing a dynamic learning mechanism that adjusts the magnitude of policy updates based on model uncertainty.

1. Measuring Uncertainty via Semantic Entropy

Unlike standard Shannon entropy, which operates at the token level, SEED-GRPO utilizes Semantic Entropy. The process involves sampling multiple responses for a given prompt and clustering them based on their semantic meaning (e.g., "x=3" and "The value of x is 3" are grouped together). High semantic entropy indicates that the model is generating many semantically distinct answers, reflecting high internal uncertainty.

2. Uncertainty-Aware Advantage Modulation

The core innovation lies in the modulation of the "advantage" signal—the metric determining how much better an answer is than the group average. SEED-GRPO applies a modulation function $f$ to the advantage:

  • Low Uncertainty: The model receives a full update signal, reinforcing correct reasoning paths.
  • High Uncertainty: The update magnitude is downscaled, resulting in a more conservative learning step.

This creates an implicit automated curriculum where the model focuses its learning on "comfort zone" problems it is close to mastering, while treading carefully on problems beyond its current knowledge boundaries.

The SEED-GRPO workflow: Rollout samples are clustered to compute semantic entropy, which then modulates the advantage function for policy optimization.

Experimental Results

The researchers evaluated SEED-GRPO on high-difficulty mathematical benchmarks, including AIME24, AMC, MATH500, Minerva, and OlympiadBench.

Performance Gains

SEED-GRPO demonstrated substantial improvements across all metrics:

  • Average Accuracy: Using a Qwen2.5-7B base, SEED-GRPO achieved an average score of 56.6%, a 5.2% improvement over the strong Dr.GRPO baseline.
  • Olympiad-Level Reasoning: On the AIME24 benchmark, the model reached 56.7% accuracy (with 16 rollouts), surpassing several 32B models such as SRPO and DAPO.
  • State-of-the-Art Distillation: When applied to the DeepSeek-R1-Distill-Qwen-7B model, SEED-GRPO achieved a new SOTA average of 64.0%, including a remarkable 91.6% on the MATH benchmark.

Discussion and Key Insights

The success of SEED-GRPO underscores several critical insights for LLM training:

  • Semantics Matter: For reasoning tasks, the semantic consistency of answers is a more reliable measure of uncertainty than token-level distributions.
  • Conservative Updates Prevent Drift: In RL, aggressive updates on high-uncertainty prompts often introduce noise. A cautious approach prevents the policy from drifting into unstable territory.
  • Efficiency over Scale: The ability of a 7B model to outperform 32B competitors suggests that algorithmic refinements in how we train are becoming as impactful as increasing model parameters.

Limitations and Future Work

While effective, SEED-GRPO currently focuses on final answer clustering, potentially missing nuances in intermediate reasoning steps. Future work may involve using external models (like RoBERTa or GPT-4o) to cluster complex reasoning paths or extending the framework to non-mathematical domains where "meaning" is more subjective. Additionally, semantic entropy could be utilized at inference time to trigger self-correction mechanisms.

Summary updated

Abstract

Group Relative Policy Optimization (GRPO) introduces a new paradigm for reinforcement learning in Large Language Models (LLMs), modifying PPO by eliminating the value model for efficient post-training. However, vanilla GRPO assigns equal weight to all prompts during policy updates, ignoring that supervision whose target answers are inconsistent with the model’s existing parameter knowledge can increase hallucinations and degrade downstream performance. To address this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which explicitly measures LLMs’ uncertainty and uses it to modulate the learn- ing process. This enables conservative updates for high-uncertainty prompts (e.g., beyond model knowledge) while preserving relatively higher signals for confident ones. Experimental results on five mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0) and on four few-shot fine-grained image classification datasets demonstrate that SEED- GRPO achieves new state-of-the-art performance in average accuracy. The code, implementation details will be publicly released.
BibTeX
@misc{chen2025seed,
  author = {Minghan Chen and Guikun Chen and Wenguan Wang and Yi Yang},
  title = {{SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization}},
  year = {2025},
  eprint = {2505.12346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.12346},
}
Figure