Abstract
Reinforcement Learning (RL) has become a cornerstone for enhancing the reasoning capabilities of Large Language Models (LLMs). However, standard algorithms like Group Relative Policy Optimization (GRPO) treat all training prompts with equal weight, ignoring the inherent uncertainty in model responses. We introduce SEED-GRPO (Semantic Entropy EnhanceD GRPO), a novel framework that integrates semantic entropy to modulate policy updates. By scaling down updates for questions where the model exhibits high inconsistency, SEED-GRPO improves training stability and reasoning performance. Our results show that a 7B parameter model trained with SEED-GRPO can outperform much larger 32B models on challenging mathematical benchmarks like AIME24.
Introduction
The recent success of models like DeepSeek-R1 and OpenAI’s o1 highlights the power of Reinforcement Learning in boosting logical reasoning. A popular approach is Group Relative Policy Optimization (GRPO), which evaluates multiple sampled answers for a single prompt to calculate relative advantages. Despite its efficiency, vanilla GRPO suffers from a "blind spot": it does not distinguish between a model that is confident and one that is merely lucky.
When an LLM provides wildly different answers to the same problem, it signals high uncertainty. Standard GRPO ignores this signal, potentially leading the model to overfit to noisy rewards or "lucky" correct guesses. This lack of nuance can result in unstable training trajectories and suboptimal reasoning capabilities.
Methodology: Uncertainty-Aware Training
SEED-GRPO addresses these limitations by introducing a dynamic learning mechanism that adjusts the magnitude of policy updates based on model uncertainty.
1. Measuring Uncertainty via Semantic Entropy
Unlike standard Shannon entropy, which operates at the token level, SEED-GRPO utilizes Semantic Entropy. The process involves sampling multiple responses for a given prompt and clustering them based on their semantic meaning (e.g., "x=3" and "The value of x is 3" are grouped together). High semantic entropy indicates that the model is generating many semantically distinct answers, reflecting high internal uncertainty.
2. Uncertainty-Aware Advantage Modulation
The core innovation lies in the modulation of the "advantage" signal—the metric determining how much better an answer is than the group average. SEED-GRPO applies a modulation function $f$ to the advantage:
- Low Uncertainty: The model receives a full update signal, reinforcing correct reasoning paths.
- High Uncertainty: The update magnitude is downscaled, resulting in a more conservative learning step.
This creates an implicit automated curriculum where the model focuses its learning on "comfort zone" problems it is close to mastering, while treading carefully on problems beyond its current knowledge boundaries.
Experimental Results
The researchers evaluated SEED-GRPO on high-difficulty mathematical benchmarks, including AIME24, AMC, MATH500, Minerva, and OlympiadBench.
Performance Gains
SEED-GRPO demonstrated substantial improvements across all metrics:
- Average Accuracy: Using a Qwen2.5-7B base, SEED-GRPO achieved an average score of 56.6%, a 5.2% improvement over the strong Dr.GRPO baseline.
- Olympiad-Level Reasoning: On the AIME24 benchmark, the model reached 56.7% accuracy (with 16 rollouts), surpassing several 32B models such as SRPO and DAPO.
- State-of-the-Art Distillation: When applied to the DeepSeek-R1-Distill-Qwen-7B model, SEED-GRPO achieved a new SOTA average of 64.0%, including a remarkable 91.6% on the MATH benchmark.
Discussion and Key Insights
The success of SEED-GRPO underscores several critical insights for LLM training:
- Semantics Matter: For reasoning tasks, the semantic consistency of answers is a more reliable measure of uncertainty than token-level distributions.
- Conservative Updates Prevent Drift: In RL, aggressive updates on high-uncertainty prompts often introduce noise. A cautious approach prevents the policy from drifting into unstable territory.
- Efficiency over Scale: The ability of a 7B model to outperform 32B competitors suggests that algorithmic refinements in how we train are becoming as impactful as increasing model parameters.
Limitations and Future Work
While effective, SEED-GRPO currently focuses on final answer clustering, potentially missing nuances in intermediate reasoning steps. Future work may involve using external models (like RoBERTa or GPT-4o) to cluster complex reasoning paths or extending the framework to non-mathematical domains where "meaning" is more subjective. Additionally, semantic entropy could be utilized at inference time to trigger self-correction mechanisms.
Summary updated