Introduction
Reinforcement Learning (RL) has become a cornerstone for aligning Large Language Models (LLMs) with complex human reasoning tasks. However, RL training is notoriously unstable due to high variance in policy updates. To mitigate this, algorithms typically employ an advantage function, which measures the relative benefit of a specific action compared to a baseline.
While traditional methods like Proximal Policy Optimization (PPO) utilize a separate, computationally expensive "Value Model" (critic network) to predict this baseline, recent innovations like Group Relative Policy Optimization (GRPO) have simplified the process. GRPO uses the mean reward of a sampled group as a baseline. However, this "naive" estimator often fails in non-stationary environments where reward signals are noisy and evolve as the model learns, leading to unstable updates and suboptimal convergence.
Methodology: Kalman Filter Enhanced GRPO (KRPO)
This paper introduces Kalman Filter Enhanced Group Relative Policy Optimization (KRPO). KRPO treats observed rewards not as absolute truths, but as noisy measurements of an underlying "true" reward baseline. By implementing a one-dimensional Kalman Filter, KRPO dynamically tracks this baseline and its associated uncertainty.
Key Innovations
- Dynamic Baseline Estimation: The filter maintains a posterior estimate of the reward baseline, adapting to the non-stationary nature of the training process.
-
Uncertainty-Aware Normalization: KRPO scales policy updates based on the estimated variance ($P_{i|i}$). The advantage is calculated as: $$A_i = \frac{r_i - \hat{x}_{i|i}}{\sqrt{P_{i|i} + \epsilon}}$$ This ensures that updates are more cautious when the reward signal is highly uncertain.
-
Efficiency: KRPO is non-parametric and requires no additional neural network components, making it a lightweight drop-in replacement for the standard GRPO advantage calculation. Training overhead is negligible, with KRPO requiring approximately 2.288 seconds per question compared to 2.271 seconds for GRPO.
Experimental Results
The researchers evaluated KRPO against standard GRPO and PPO across several mathematical reasoning benchmarks, including Arithmetic (100,000 problems) and OpenMath-Instruct.
Performance on Arithmetic Tasks
In arithmetic benchmarks ranging from "Easy" to "Hard," KRPO demonstrated superior stability and faster convergence compared to the baseline GRPO.
As shown in Figure 1, KRPO consistently reaches higher return levels earlier in the training process. On the "Normal" arithmetic set, KRPO achieved a 5.361% improvement in accuracy over standard GRPO.
Advanced Mathematical Reasoning
The benefits of KRPO are most pronounced in complex reasoning tasks. On the "Hard" subset of OpenMath-Instruct, KRPO delivered a significant accuracy boost of 17.876% over standard GRPO.
Figure 2 illustrates the substantial gap in returns between KRPO and GRPO on high-difficulty math problems. Qualitative analysis suggests that KRPO-trained models are better at maintaining logical consistency throughout multi-step derivations, whereas GRPO models often falter during variable substitution or complex calculations.
Conclusion and Future Work
KRPO offers a principled, statistically grounded approach to stabilizing RL for LLMs without the computational burden of a separate critic model. By treating the reward baseline as a dynamic state estimated via a Kalman Filter, KRPO achieves significant performance gains and faster convergence on mathematical reasoning benchmarks.
Future research will explore the application of KRPO to subjective tasks (e.g., summarization) where reward signals from Reward Models are inherently noisier. Additionally, investigating adaptive noise parameters ($Q$ and $R$) could further enhance the filter's robustness across diverse domains.
Summary updated