TL;DR
Group Relative Policy Optimization (GRPO) treats all training prompts equally, forcing the model to learn aggressively even on problems beyond its existing knowledge base. SEED-GRPO solves this by estimating prompt-level epistemic uncertainty via semantic entropy—computed directly from GRPO's sampled rollouts at virtually zero extra computational cost—and dynamically dampening policy updates on high-uncertainty prompts. This acts as an automatic curriculum, boosting mathematical reasoning (e.g., jumping from 43.3% to 56.7% on AIME24 for a 7B model) and fine-grained visual recognition.
What Problem Does This Paper Address?
Reinforcement learning (RL) post-training algorithms like GRPO eliminate value networks by sampling multiple rollouts per prompt and optimizing against group-relative advantages. However, standard GRPO assigns uniform importance to every prompt.
Prior work shows that heavily updating a model on supervision conflicting with its pre-existing parametric knowledge increases hallucinations, slows convergence, and harms downstream generalization. Forcing strong gradient updates on questions the model cannot yet reliably comprehend risks overfitting to noisy reward signals. The authors seek a principled way to detect when a prompt lies outside the model's knowledge boundary and calibrate policy updates accordingly.
Methods
SEED-GRPO introduces uncertainty-aware advantage modulation into the GRPO framework:
-
"Free Lunch" Semantic Entropy ($SE$): GRPO already generates $N$ rollout responses $\{o_1, \dots, o_N\}$ per prompt $q$ to compute baseline-subtracted advantages. SEED-GRPO clusters these responses by semantic meaning (in math, grouping by equivalent final boxed answers into $K$ clusters) and computes the Shannon entropy across the resulting cluster probability distribution: $$\text{SE}(q) \approx -\sum_{k=1}^K \hat{P}(C_k \mid q) \log \hat{P}(C_k \mid q)$$ Because the rollouts are already generated for advantage estimation, calculating this prompt-level uncertainty requires virtually no extra inference overhead.
-
Advantage Modulation: The scalar advantage $A_i$ for each response is scaled by an uncertainty-dependent modulation function: $$\hat{A}_i = A_i \cdot f\left(\alpha \cdot \frac{\text{SE}(q)}{\text{SE}_{\max}}\right)$$ where $\text{SE}_{\max} = \ln N$ is the maximum theoretical entropy, $\alpha$ tunes sensitivity, and $f(\cdot)$ scales down advantages when uncertainty is high. The default choice is a linear penalty: $f_{\text{linear}}(\tilde{s}) = 1 - \alpha \tilde{s}$.
-
Implicit Curriculum Learning: By scaling advantages down for high-uncertainty prompts, the effective learning rate becomes prompt-adaptive. The model takes confident updates on problems it understands while taking conservative steps on ambiguous, out-of-boundary questions.
Datasets
- Training: The Level 3–5 subset of the MATH benchmark (~8.5k problems).
- Mathematical Evaluation:
- AIME24: 30 high-school olympiad problems from the 2024 American Invitational Mathematics Examination.
- AMC: 83 intermediate-difficulty math problems.
- MATH500: 500 challenging problems spanning algebra, geometry, and number theory.
- Minerva (MIN): 272 multi-step quantitative reasoning questions.
- OlympiadBench (OLY): 675 high-difficulty olympiad-level questions.
- Multimodal Evaluation: 4-shot fine-grained image classification on Flowers102, Pets37, FGVC Aircraft, and Cars196.
- Note: No new datasets were introduced; the authors focus on standard reasoning and vision benchmarks.
Key Insights and Ideas
- Zero-overhead uncertainty estimation: Typical uncertainty quantification methods require separate ensembles or extra forward passes. SEED-GRPO repurposes the group rollouts that GRPO must generate anyway, turning advantage estimation and uncertainty modeling into a symbiotic operation.
- More rollouts produce dual benefits: Scaling rollouts from $G=8$ to $G=16$ not only refines relative advantage estimates but also sharpens semantic clustering accuracy, driving large jumps on tough benchmarks like AIME24 (46.7% $\to$ 56.7%).
- Reversing uncertainty weighting triggers collapse: In control experiments where certain prompts were downweighted and uncertain prompts were prioritized, training quickly degraded and suffered complete model collapse around step 600, proving that learning aggressively on high-uncertainty prompts actively harms the policy.
- Linear penalization is best: Comparing linear, exponential, and focal-style weight decay functions revealed that simple linear modulation provides the most stable and balanced performance across diverse benchmarks.
Key Results
- Math Reasoning (Qwen2.5-Math-7B Base):
- Standard SEED-GRPO ($G=8$) achieves a 56.6% average accuracy across five benchmarks—a +5.2% improvement over Dr.GRPO 7B (51.4%).
- With $G=16$, performance rises to 58.2% average, scoring 56.7% on AIME24 and 83.4% on MATH, outperforming several 32B-parameter models including SRPO (44.3% AIME), DAPO (50.0% Avg@32), and DeepSeek-R1-Zero-Qwen-32B (46.7% AIME).
- Reasoning Distillation (DeepSeek-R1-Distill-Qwen-7B):
- Reaches 64.0% average across benchmarks, and 67.4% average (63.3% AIME24, 93.2% MATH) when trained with an 8k context length.
- Multimodal Generalization (Qwen2-VL-2B):
- Reaches 88.5% 4-shot average accuracy on fine-grained image classification, consistently outperforming supervised fine-tuning (55.6%), vanilla GRPO (81.9%), and GPG (86.0%).
Limitations and Future Work
- Final-Answer Simplification: Semantic clustering currently checks only final boxed answers. It ignores intermediate reasoning trajectories, potentially penalizing valid alternative derivation strategies or failing on open-ended problems without unique deterministic answers.
- Clustering Robustness: The method depends on the accuracy of answer parsing; unparseable or diverse text outputs in subjective tasks pose challenges for exact semantic grouping.
- Future Directions: Incorporating step-level intermediate reasoning into semantic entropy estimation, adapting the method to code generation and open-ended QA, and exploring semantic entropy as an inference-time routing/fallback mechanism.
Summary updated