Parameter Exploration for RLVR via Variational Learning

Vatsal Venkatkrishna, Nico Daheim, Iryna GurevycharXiv 2026 · arXiv

TL;DR

This paper introduces Perturbed Parameter Policy Optimization (3PO), a novel framework that shifts LLM exploration from the action space (tokens) to the parameter space (weights) during Reinforcement Learning with Verifiable Rewards (RLVR). By sampling diverse policies from a learned approximate posterior using the IVON optimizer, 3PO discovers high-quality reasoning paths that standard action-space methods miss. The approach consistently improves mathematical reasoning and code generation performance across 7B models at a near-identical FLOPs cost.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm for boosting LLM reasoning capabilities in domains like math and code, where answers can be automatically verified. Popular algorithms like Group-Relative Policy Optimization (GRPO) rely on generating a group of rollouts for a prompt and calculating rewards relative to the group average.

However, these methods frequently suffer from stalled learning. If an LLM is not yet capable enough, all rollouts in a group might fail (or succeed), resulting in a "zero-advantage group" where no learning signal is generated.

To force exploration, practitioners typically turn up the softmax temperature during generation. But temperature scaling has a fundamental flaw: it only flattens the token distribution variance; it is rank-preserving and cannot reorder which tokens are most likely. Turn it up too high, and the model simply generates incoherent, malformed gibberish rather than novel, correct reasoning paths.

Figure 1: Temperature versus Weight noise. Temperature scaling controls the entropy of the token distribution but preserves relative token ordering. Adding noise to the model’s weights can completely alter the token distribution, helping the model explore new regions of the solution space.

Methods

Instead of adding noise to the output tokens (action space), the authors propose adding noise directly to the model's weights (parameter space) using variational learning. They utilize IVON, a natural-gradient-based optimizer that learns a Gaussian distribution (mean and diagonal variance) over the network parameters.

The authors introduce three distinct noising strategies under the 3PO umbrella:

  1. B3PO (Batched 3PO): Samples a single weight perturbation per gradient step and reuses it across all prompts in a batch.
  2. M3PO (Multi-sample 3PO): Samples $M$ independent weight perturbations, generates rollouts for each, and accumulates the loss before performing a single weight update to reduce gradient variance.
  3. C3PO (Chunked 3PO): The most sophisticated variant. It divides a single GRPO rollout group across $N$ independently sampled weight configurations. Advantages are calculated across the entire aggregated group, maximizing the diversity of trajectories within a single advantage calculation.

To ensure training stability against the training-inference mismatch introduced by multi-model sampling, the authors incorporate sequence-level importance sampling (Seq-MIS).

Overview of noising strategies

Datasets

The methods were trained and evaluated using two model families: OLMo-3-1025-7B and Qwen2.5-Math-7B.

  • Training Datasets:
  • Math: DAPO-MATH-17k (~17,000 math problems with ground-truth answers).
  • Code: CodeR1-12K (verified coding problems from LeetCode and TACO).
  • Evaluation Benchmarks:
  • Math: AIME (2024–2026), AMC 2023, MATH-500, and Minerva.
  • Code: LiveCodeBench-v6 (problems released between January and April 2025).

Key Insights and Ideas

  • Token Reordering via Weight Noise: Unlike temperature scaling, perturbing weights alters the underlying logit values dynamically. This allows the model to completely reorder token distributions, unlocking entirely new solution paths without causing text degeneration.
  • Rescuing "Dead" Groups: 3PO successfully "rescues" zero-advantage groups during training. It generates at least one correct rollout where standard GRPO yields none, sustaining a healthy learning signal throughout the training run.
  • Task Hardness Correlation: Parameter-space exploration provides the most significant benefits on the hardest tasks (e.g., AIME and LiveCodeBench). When the base model's initial capability is low, action-space exploration fails, but parameter noise successfully uncovers rare, high-reward trajectories.

Key Results

  • Downstream Performance Boosts: C3PO achieved the highest average performance across both model families. On the challenging AIME benchmarks, C3PO outperformed standard GRPO by up to 4.17% on OLMo-3 and 5.83% on Qwen2.5-Math.
  • Code Generation Gains: On LiveCodeBench, C3PO achieved a dominant accuracy curve, reaching the final reward threshold of GRPO in less than half the training steps.
  • Fewer Invalid Rollouts: Compared to action-space baselines like Polaris or entropy regularization, 3PO variants consistently produced fewer malformed or repetitive rollouts during training.
  • Compute Efficiency: The algorithmic FLOPs cost remains near-identical to standard GRPO because the elementwise noise operations are computationally trivial.

Limitations and Future Work

  • Wall-Clock Overhead: Despite identical FLOPs, 3PO variants are currently $\approx 1.5\times$ slower in wall-clock time. This is a systems-level bottleneck caused by vLLM's lack of optimized support for concurrent multi-model sampling, requiring separate inference operations.
  • Scale Constraints: The study was limited to 7B parameter models due to compute constraints. The authors hypothesize that larger models will tolerate even more exploration before training collapses.
  • Simpler Posteriors: The authors utilized a diagonal Hessian approximation via IVON. Exploring structured, non-diagonal posterior estimators could yield even more directed and effective exploration strategies.

Summary updated

Abstract

Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.
BibTeX
@misc{venkatkrishna2026parameter,
  author = {Vatsal Venkatkrishna and Nico Daheim and Iryna Gurevych},
  title = {{Parameter Exploration for RLVR via Variational Learning}},
  year = {2026},
  eprint = {2608.09805},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.09805},
}
Figure