Synthetic Control Arms for Reward Edits in Multi-Objective RL

When practitioners fine-tune a language model with reinforcement learning against several rewards at once (say correctness, format, and brevity), they routinely change the reward partway through. They add a length penalty, reweight a format bonus, or switch normalization schemes. They then judge the change by comparing the model before and after on a few benchmarks. In clinical research this design has a name, the uncontrolled single-arm pre/post study, and it is known to mislead. Without a group that did not receive the intervention, you cannot tell the effect of the treatment apart from what would have happened anyway. The same problem appears here. Continued training drifts the model on its own, other reward terms keep acting on it, and checkpoints are noisy. The consequence is concrete. Reward Bias Substitution shows that adding a length penalty to a Llama-3.2-3B model trained with GRPO (group-relative policy optimization, an RL method that scores each sampled answer against the others for the same prompt) shortened answers but raised calibration error from 0.25 to 0.41 and cut TriviaQA accuracy from 56% to 42%. The paper argues that static audit benchmarks cannot separate real debiasing from pressure that has moved onto another feature. It also reports that none of 25 surveyed debiasing papers measured the off-target features on the policy's own outputs.

This proposal borrows the causal-inference answer to missing control groups. For a reward edit made at training step $T_0$, we build a synthetic control arm: a weighted combination of previously logged "donor" training runs that did not receive the edit. The weights are fitted so the combination matches the edited run's trajectory across many behavioral metrics before $T_0$. After $T_0$, the gap between the edited run and its synthetic twin estimates the causal effect of the edit on every metric, including metrics nobody targeted. We then test whether this cheap, reusable counterfactual agrees with the expensive gold standard, a real concurrent control run. If it does, the synthetic arm can serve as the default audit for multi-objective reward changes.

Core Claim

Our claim is that off-target effects of reward edits, which is where bias substitution lives, can be estimated accurately from a library of existing runs. The fit uses multi-metric synthetic control from mRSC: Multi-dimensional Robust Synthetic Control. That method builds a counterfactual for a treated unit as a combination of untreated donors, and it does so after denoising a matrix that stacks $K$ metrics side by side. Its guarantee is that error shrinks roughly as $1/\sqrt{K T_0}$, so extra metrics act like extra pre-intervention history. That scaling is what makes the approach plausible for training runs. The pre-edit window is often short, but we can log many metrics cheaply at every checkpoint.

For edited run $u$, metric $k$, and checkpoint $t > T_0$, the estimated effect is

$$\hat{\tau}_{k,t} = y_{u,k,t} - \sum_{j} \hat{\beta}_j \, y_{j,k,t},$$

where $y_{j,k,t}$ is donor run $j$'s value of metric $k$ at step $t$. The weights $\hat{\beta}$ are fitted jointly on all $K$ metrics over steps $t \le T_0$ after low-rank denoising. A substitution is flagged when an untargeted metric's $\hat{\tau}$ leaves the band traced out by placebo runs, meaning donors to which we apply a fake edit date.

Three bodies of work shape the design. The substitution paper supplies both the motivation and the failure we must detect. mRSC supplies the estimator, along with its rank diagnostic: if adding a metric raises the rank of the denoised matrix, that metric has discordant drivers and should not be pooled. The multi-objective methods (GDPO, ODIN, HAPO) are the interventions under audit.

Each of these methods fixes one axis. GDPO normalizes each reward separately. ODIN removes length from the reward model with a second head and a decorrelation loss. HAPO rewards beating each problem's shortest correct answer so far. None of them measures what its change does to the axes it did not target, relative to a counterfactual.

Relation to the Closest Work

The nearest bridge in the literature is Reward Shaping Control Variates. It uses potential-based shaping to cut variance in off-policy evaluation of clinical policies from logged data. It evaluates a fixed policy's scalar return, and it does not study how a training-time change shifts many behaviors. Multi-Task Adversarial Learning for Treatment Effect Estimation in Basket Trials recovers counterfactuals for uncontrolled cohorts by sharing a representation across tumor types. It is a static regression and has no notion of a training trajectory.

Our proposal differs from both in two ways. The treated units are training runs, and the outcome is a panel of behavioral metrics on the policy's own outputs. We make no claim to improve optimization itself. The contribution is a measurement instrument, validated against real controls.

Research Program

The first component is a donor library. Starting from shared warm-up checkpoints, we run many GRPO trainings that vary reward weights, which reward terms are present, and seeds. Every run uses the same prompt order, so that shocks caused by particular data batches are common to all runs.

At every checkpoint we log a panel of roughly 10 to 15 metrics on policy-generated outputs from a fixed probe set. The panel covers response length, expected calibration error, TriviaQA accuracy, hedging and assertion-strength frequency, n-gram repetition, format validity, task accuracy, and scores from the reward model and from a held-out judge.

The second component is the estimator. We apply mRSC to these panels, run the rank diagnostic when choosing which metrics to pool, and build placebo bands from donors that received no edit.

The third component is validation. For each audited edit we also train paired concurrent control arms. These are the same run with the edit withheld, over several seeds, sharing prompts and sampling seeds with the edited run. Their difference serves as the ground-truth effect.

The fourth component applies the validated instrument. We use it to ask whether GDPO, ODIN-style disentanglement, HAPO, and plain length penalties cause substitution. We also test the tension, raised by Visionary-R1, between conciseness rewards and the long, descriptive traces that keep a model grounded in the image. Visionary-R1 tied grounding to explicit caption rewards and to longer reasoning.

Baselines

The naive reference point is the uncontrolled pre/post comparison: the metric at the end minus the metric at $T_0$ on the edited run. This is what most papers effectively report. Our method beats it if its effect estimates have substantially lower error against the paired-control ground truth. We predict it will attribute ordinary training drift to the edit.

The second baseline is interrupted time-series extrapolation. It fits a trend to each metric on the edited run before $T_0$ and projects that trend forward, using no donors. It is the strongest single-run method, and it isolates what donors add. Its budget is identical to ours: the same checkpoints and probe set.

The third baseline is the static-benchmark audit criticized by the substitution paper, which compares ranking accuracy or win rate on a fixed audit set. The substitution paper argues it cannot detect bias substitution. Its reported case, where TriviaQA fell from 56% to 42% and calibration error rose from 0.25 to 0.41 under a length penalty, is our first replication target. The instrument must flag that case, and the static audit should miss it.

The fourth comparison is an ablation: single-metric robust synthetic control, fitted one metric at a time. It isolates the multi-metric pooling. In its own paper, mRSC reports 5–7x lower test prediction error than single-metric synthetic control on Walmart sales when the pre-period is 10% of the timeline. We have found no reported result on Lacuna for synthetic control on RL training trajectories, so whether that gain carries over is an open empirical question.

The gold standard is the paired concurrent control. It is not a competitor to beat. It is the target to match at lower compute.

Across all arms, donor runs, edited runs, and controls share the model, data order, rollout count, and tuning budget.

Why not just…?

Why not just run a real control arm every time? This is the serious threat, and we treat it as ground truth rather than an opponent. For a single edit, a paired run with common random numbers is affordable and probably better. The case for a synthetic arm is cost when many edits are compared. Leave-one-out and interaction arms multiply with the number of reward terms, and seed variance means each arm needs several runs, while a donor library is paid for once. We will report compute-matched accuracy. If the synthetic arm needs as many new runs as paired controls to reach the same error, the honest conclusion is to recommend paired controls with a mandatory off-target panel, and we will say so.

Why not just evaluate on more static benchmarks? The substitution paper's argument is that fixed audit sets cannot separate real debiasing from substitution. More benchmarks do not create a counterfactual for the policy's own shifting output distribution. The static-audit baseline above tests this directly.

Why not just extrapolate the run's own pre-edit trend? Training dynamics bend: accuracy under GRPO collapses around 400 steps in GDPO's analysis. Trend extrapolation cannot anticipate such shifts, while donors trained on the same data order share them. The interrupted time-series baseline measures how much this matters. If it matches our method, donors are unnecessary.

Why not just use a disentangled reward like ODIN? ODIN removes one known confound, length, from the reward model. Substitution by construction moves onto features nobody named. Disentangling is an intervention that needs auditing, not a replacement for the audit, and we include it among the audited edits.

Proposed Experiments

The first setting replicates the substitution paper: Llama-3.2-3B trained with GRPO on UltraFeedback, with a length penalty added at $T_0$. We measure length, calibration error, and TriviaQA accuracy on policy outputs.

The second setting follows GDPO: DeepSeek-R1-1.5B trained on DeepScaleR-Preview. Edits include conciseness rewards under GRPO summation, GDPO's decoupled normalization, and HAPO's history-based length reward. Evaluation uses AIME-24 and MATH500. On AIME-24, GDPO reports up to +6.3% accuracy over GRPO, and HAPO reports 49–59% token reduction with roughly 2% accuracy loss.

The third setting follows GDPO's tool-calling experiments: Qwen2.5-1.5B on BFCL-v3, where GDPO reports +5% live accuracy and +4% format compliance over GRPO. The question here is whether format gains shift other behaviors.

The primary metric is the mean absolute error of $\hat{\tau}_{k,t}$ against the seed-averaged paired-control effect, for each metric, computed at matched extra compute. Secondary metrics are the area under the ROC curve for flagging substitution events identified by the controls, and false-flag rates from placebo edits.

Three checks are decisive. First, the synthetic arm must flag the length-penalty substitution that the static audit misses. Second, it must beat interrupted time-series and single-metric synthetic control on effect error when the pre-period is short. Third, its error must fall as metrics are added, roughly as the $1/\sqrt{K}$ form inherited from mRSC predicts, except when the rank diagnostic warns that a metric is discordant.

If all three hold, the instrument lets us report, for each of GDPO, ODIN, HAPO, and length penalties, where its optimization pressure goes. If the first check fails, the synthetic arm cannot stand in for real controls on the failure that motivates it.