Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning

Hu Wang, Congbo Ma, Ian D. Reid, Mohammad YaqubarXiv 2025 · DOI · arXiv

Introduction

Reinforcement Learning (RL) has become a cornerstone for aligning Large Language Models (LLMs) with complex human reasoning tasks. However, RL training is notoriously unstable due to high variance in policy updates. To mitigate this, algorithms typically employ an advantage function, which measures the relative benefit of a specific action compared to a baseline.

While traditional methods like Proximal Policy Optimization (PPO) utilize a separate, computationally expensive "Value Model" (critic network) to predict this baseline, recent innovations like Group Relative Policy Optimization (GRPO) have simplified the process. GRPO uses the mean reward of a sampled group as a baseline. However, this "naive" estimator often fails in non-stationary environments where reward signals are noisy and evolve as the model learns, leading to unstable updates and suboptimal convergence.

Methodology: Kalman Filter Enhanced GRPO (KRPO)

This paper introduces Kalman Filter Enhanced Group Relative Policy Optimization (KRPO). KRPO treats observed rewards not as absolute truths, but as noisy measurements of an underlying "true" reward baseline. By implementing a one-dimensional Kalman Filter, KRPO dynamically tracks this baseline and its associated uncertainty.

Key Innovations

  1. Dynamic Baseline Estimation: The filter maintains a posterior estimate of the reward baseline, adapting to the non-stationary nature of the training process.
  2. Uncertainty-Aware Normalization: KRPO scales policy updates based on the estimated variance ($P_{i|i}$). The advantage is calculated as: $$A_i = \frac{r_i - \hat{x}_{i|i}}{\sqrt{P_{i|i} + \epsilon}}$$ This ensures that updates are more cautious when the reward signal is highly uncertain.

  3. Efficiency: KRPO is non-parametric and requires no additional neural network components, making it a lightweight drop-in replacement for the standard GRPO advantage calculation. Training overhead is negligible, with KRPO requiring approximately 2.288 seconds per question compared to 2.271 seconds for GRPO.

Experimental Results

The researchers evaluated KRPO against standard GRPO and PPO across several mathematical reasoning benchmarks, including Arithmetic (100,000 problems) and OpenMath-Instruct.

Performance on Arithmetic Tasks

In arithmetic benchmarks ranging from "Easy" to "Hard," KRPO demonstrated superior stability and faster convergence compared to the baseline GRPO.

Training curves on arithmetic questions of varying difficulty.

As shown in Figure 1, KRPO consistently reaches higher return levels earlier in the training process. On the "Normal" arithmetic set, KRPO achieved a 5.361% improvement in accuracy over standard GRPO.

Advanced Mathematical Reasoning

The benefits of KRPO are most pronounced in complex reasoning tasks. On the "Hard" subset of OpenMath-Instruct, KRPO delivered a significant accuracy boost of 17.876% over standard GRPO.

Performance comparison on hard math questions.

Figure 2 illustrates the substantial gap in returns between KRPO and GRPO on high-difficulty math problems. Qualitative analysis suggests that KRPO-trained models are better at maintaining logical consistency throughout multi-step derivations, whereas GRPO models often falter during variable substitution or complex calculations.

Conclusion and Future Work

KRPO offers a principled, statistically grounded approach to stabilizing RL for LLMs without the computational burden of a separate critic model. By treating the reward baseline as a dynamic state estimated via a Kalman Filter, KRPO achieves significant performance gains and faster convergence on mathematical reasoning benchmarks.

Future research will explore the application of KRPO to subjective tasks (e.g., summarization) where reward signals from Reward Models are inherently noisier. Additionally, investigating adaptive noise parameters ($Q$ and $R$) could further enhance the filter's robustness across diverse domains.

Summary updated

Abstract

The advantage function is a central concept in RL that helps reduce variance in policy gradient estimates. For language modeling, Group Relative Policy Optimization (GRPO) was proposed to use the within-group sample mean as a baseline for advantage normalization. This estimator can be sensitive to small group size and rollout-level stochasticity, which may lead to suboptimal advantage estimates in some settings. In this paper, we propose Kalman Filter Enhanced Group Relative Policy Optimization (KRPO), a lightweight variant that treats per-group rewards as noisy observations of a latent prompt-level reward baseline and uses a 1D Kalman filter to estimate both the baseline and its uncertainty. KRPO introduces no additional learned parameters and can be integrated into GRPO with minimal computational overhead. On mathematical reasoning benchmarks, KRPO consistently improves training reward curves and final accuracy over GRPO. These results suggest that adaptive advantage estimation is a promising direction for critic-free reinforcement learning in language model reasoning. The code is available at https://github.com/billhhh/KRPO_LLMs_RL.
BibTeX
@misc{wang2025kalman,
  author = {Hu Wang and Congbo Ma and Ian D. Reid and Mohammad Yaqub},
  title = {{Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning}},
  year = {2025},
  eprint = {2505.07527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.07527},
}
Figure