Reinforcement learning for language models wastes most of what it generates. On-policy methods such as Group Relative Policy Optimization (GRPO) train only on reasoning traces sampled by the current model. GRPO scores each sampled answer against the other answers drawn for the same prompt. Each expensive trace is used for one update and then thrown away. Recent systems recover this waste by keeping a replay buffer, a store of past traces. They are selective about what they keep: past successes, prompts the model once failed, traces of medium difficulty and low entropy. This note argues that such a buffer has the same structure as a retrospective cohort study in epidemiology. Its records were produced by earlier "practitioners" (old checkpoints of the model), and they entered the dataset through a selection process that depends on outcomes. Epidemiology has a mature toolkit for exactly this situation: inverse-probability-of-selection weights, weight trimming, and balance diagnostics between the sampled cohort and the target population. We propose to bring that toolkit into replay-based reinforcement learning for language models, and to measure directly how much of replay's benefit, and how much of its known failure mode of collapsing diversity, comes from selection bias rather than from reuse itself.
The core claim fits in two sentences. A success-only or difficulty-filtered buffer gives a gradient that is not the policy gradient. It is a gradient conditioned on how traces were selected, and existing replay methods correct only for the change in token probabilities between old and new model, not for this selection. Estimating each replayed trace's probability of having entered the buffer, and reweighting by it with epidemiology-style trimming, should keep most of replay's savings in generated rollouts while reducing entropy collapse. We test this on mathematical reasoning and on a long-context clinical reasoning benchmark where rollouts are especially costly.
Core Claim
The bias has a simple form. Take a single prompt $x$ with binary reward $r$, and let $p(x)$ be the current model's success rate on it. The policy gradient for that prompt is $\mathbb{E}_{\pi}[r\,\nabla\log\pi(y|x)] = p(x)\,\mathbb{E}_{\pi}[\nabla\log\pi(y|x)\mid r=1]$. A buffer that keeps only successes and replays them uniformly drops the factor $p(x)$. In effect it weights every prompt by $1/p(x)$ relative to the true gradient, so it concentrates on prompts the model rarely solves while pushing probability mass onto the few successful traces it happened to find. This is outcome-dependent sampling in the epidemiological sense. Difficulty filters such as the 25–75% success window in ExGRPO add a second, prompt-level selection. Its probability can be computed from a binomial model of the group sampling, given an estimate of $p(x)$ under the checkpoint that generated the group. Buffer age adds a third source of shift, which is the one existing truncated importance ratios already handle.
We write the corrected replay weight for a trace $y$ on prompt $x$, generated by checkpoint $k$, as
$$w(x,y,k) = \mathrm{clip}\!\left(\frac{\pi_\theta(y|x)}{\pi_k(y|x)}\right)\cdot \mathrm{trim}\!\left(\frac{1}{\hat{s}(x,y,k)}\right),$$
where $\pi_\theta$ is the current model, $\pi_k$ the generating checkpoint, and $\hat{s}$ the estimated probability that the trace entered and survived in the buffer. The clip is the usual token-level ratio clipping. The trim caps extreme selection weights at a percentile, as in the propensity-weighting literature Weight Trimming and Propensity Score Weighting. Following Importance sampling in reinforcement learning with an estimated behavior policy, which showed in control tasks that weighting by an estimated rather than true behavior probability lowers variance, and the parallel result for estimated propensity scores Matching on the Estimated Propensity Score, we estimate $\hat{s}$ empirically. The estimator is a small calibrated model over prompt success rate, checkpoint age, trace length and trace entropy, fit on the buffer's actual inclusion history. We use it even where an analytic selection probability exists.
We do not assume the uncorrected bias is harmful. The implicit $1/p(x)$ up-weighting may be part of why Buffer Matters solves 40.7% of problems the base model consistently failed. The proposal is to make this weighting explicit and controllable, separating useful prompt reweighting from harmful collapse onto a few traces. It is not a claim that unbiasedness wins by default.
Relation to Prior Work
Three replay methods motivate the work, because they show that reuse helps. Each designs a selection rule without correcting for it. Buffer Matters introduced Batch Adaptation Policy Optimization (BAPO), which mixes fresh high-variance rollouts, re-attempted failures and buffered successes. ExGRPO buckets prompts by difficulty and replays low-entropy verified traces. Retrospective Replay restarts generation from promising intermediate prefixes. We reuse their buffers as the treatment arms and change only how replayed traces are weighted.
The off-policy correction literature supplies the method and also constrains it. Truncated importance weights and trace cutting, from ACER and Retrace, and dynamically scaled clipping in Off-Policy PPO, correct for action shift only. Their guarantees assume the replayed data are an unselected sample of the behavior policy. Outcome-filtered language-model buffers violate that assumption. Doubly Robust Policy Gradient suggests a natural extension, adding a learned value baseline to make the estimator doubly robust. Its need for gradients of the value model through simulated dynamics makes it a later step, not the core.
On the clinical side, EHR-R1 trains reasoning models on MIMIC-IV-derived patient cohorts with plain on-policy GRPO and no replay. It is the application regime. Long patient records make each rollout costly, and hard diagnostic cases produce groups where every answer fails, so the relative advantage is zero. Offline clinical reinforcement learning such as DeepVent and MeDT learns from physician-generated logs. That is a different problem, because there the behavior policy is unknown and confounded by unrecorded patient severity. In our setting the "practitioners" are model checkpoints whose probabilities we log, so the only unknown is the selection mechanism. This is where the cohort analogy is exact rather than loose, and we deliberately do not claim to solve clinical confounding.
Research Program
The first step is diagnostic and needs no new algorithm. During BAPO-style and ExGRPO-style training, we periodically draw a large fresh on-policy sample at the current checkpoint. This is the "target population." We compare it with the replayed batch using standardized mean differences, the standard cohort balance statistic. The comparison covers prompt success rate, trace length, token entropy and checkpoint age. We also compute the cosine similarity between each replay estimator's gradient and a high-sample on-policy reference gradient, on a held-out set of parameters such as one layer's LoRA adapters to keep this cheap. Together these quantify the bias that the correction targets, and show whether it grows with buffer age and selectivity.
The second step builds the selection-propensity estimator and the weighted objective above. We use three variants: analytic selection probabilities, empirically estimated ones, and estimated ones with percentile trimming at several caps. The third step makes the prompt reweighting explicit. We add a separate, tunable prompt weight $p(x)^{-\alpha}$, where $\alpha=0$ recovers the corrected policy gradient and $\alpha=1$ recovers the implicit success-only weighting. This tests whether hard-prompt focus and trace-level collapse can be separated.
Baselines
The naive reference point is plain on-policy GRPO with no replay, run at the same number of generated rollouts. This is the EHR-R1 recipe, and it is the right floor because the question is whether replay is worth its bias. A second reference is on-policy GRPO given extra rollouts until it matches each replay method's final accuracy. This measures the rollout savings directly, in the spirit of BAPO's reported 2.5x fewer rollouts than DAPO.
The strongest existing methods are BAPO and ExGRPO. BAPO reports an average 12.5% improvement over GRPO across math, planning and geometry tasks, and +5.5 points on AIME 2024 when applied to PPO, in Buffer Matters. ExGRPO reports +3.5 points on math and +7.6 points out of distribution on Qwen2.5-Math-7B in ExGRPO. We rerun both at matched settings rather than copying numbers: same base model, same training data (the DeepScaleR 40k set for math), same rollout budget, same buffer size, and an equal hyperparameter sweep of about eight configurations per method.
On the clinical side, EHR-R1-72B reports 0.6744 decision-making accuracy and 0.9328 risk-prediction AUROC on EHR-Bench. Our runs use a 7B-scale model, so those figures set context, not a target. Our comparison is GRPO versus replay variants at equal scale.
The key ablation isolates the selection correction. It is the same buffer with only the token-level clipped ratio, identical except that $\hat{s}$ is set to one. Beating it means at least one of two things at equal rollouts. One is equal or better accuracy with clearly less entropy decay and better pass@k at large k. The other is higher accuracy on the clinical benchmark's hard-case subset. Either must hold across three seeds.
Why not just…?
Why not just cap buffer age or lower the replay ratio? Keeping only recent traces shrinks action shift and would plausibly reduce collapse with no new machinery. This is a real threat. We run a sweep over age caps and replay ratios and compare our method against the best point on that frontier at equal rollouts. If a tuned age cap matches it, we will report that selection bias is not the operative problem. The balance diagnostics will show whether age or outcome selection drives the gradient bias.
Why not just rely on token-level importance ratios and PPO clipping? These correct for the change in model probabilities. They do not correct for which traces were admitted, because a success-only buffer is biased even at zero model drift. The ablation with $\hat{s}=1$ tests exactly this. The gradient-cosine measurement shows whether that variant stays biased when the checkpoint has barely moved.
Why not just add an entropy bonus? An entropy bonus treats the symptom generically. It cannot tell useful hard-prompt focus apart from trace-level overconfidence. We include a tuned entropy-bonus arm. If it matches our method, the selection-weighting story is unnecessary, and we will say so.
Why not just do offline RL or supervised fine-tuning on successful traces? Fine-tuning on successes is the $\alpha=1$, no-fresh-data extreme of the same biased objective. Retrospective Replay reports supervised fine-tuning at 28.7% on MATH against 34.3% for replay-based reinforcement learning. We include rejection-sampled fine-tuning as a reference arm.
Proposed Experiments
The math study trains a 7B base model, Qwen2.5-Math-7B for comparability with ExGRPO, on the DeepScaleR set. We evaluate on AIME24, AMC23, MATH-500, OlympiadBench and Minerva, with ARC-Challenge, GPQA-Diamond and MMLU-Pro for out-of-distribution reasoning. Llama-3.1-8B Base is a second model where ExGRPO reports that on-policy training collapses.
The clinical study uses EHR-Bench decision-making and risk-prediction tasks built from MIMIC-IV, which requires credentialed access, with Stanford EHRSHOT for zero-shot transfer. We report accuracy and AUROC against the number of generated rollouts and against generated tokens, since long records make tokens the true cost.
Across both studies the metrics are accuracy against rollouts, token entropy over training, pass@k up to k=64, gradient cosine to the on-policy reference, and effective sample size of the weights.
The decisive checks are three. First, the uncorrected buffers must show measurable gradient bias that grows with selectivity; if they do not, the premise fails. Second, the selection-corrected variant must retain most of replay's rollout savings while slowing entropy decay and improving pass@k relative to the $\hat{s}=1$ ablation and the tuned age-cap baseline. Third, on EHR-Bench, the effect should be largest on hard cases where every answer in a group fails, since that is where outcome-selected replay is most skewed.
References
- ExGRPO: Learning to Reason from Experience
- Weight Trimming and Propensity Score Weighting
- Importance sampling in reinforcement learning with an estimated behavior policy
- Matching on the Estimated Propensity Score
- Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
- Improving RL Exploration for LLM Reasoning through Retrospective Replay
- Sample Efficient Actor-Critic with Experience Replay
- Safe and Efficient Off-Policy Reinforcement Learning
- Off-Policy Proximal Policy Optimization
- From Importance Sampling to Doubly Robust Policy Gradient
- EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis
- Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning
- Empowering Clinicians with MeDT: A Framework for Sepsis Treatment