Measuring and Blocking Off-Manifold Leakage in Reward Fine-Tuning of Straight Flows

When a text-to-image generator is fine-tuned to raise an automated score, such as an OCR legibility reward or a learned aesthetic preference model, it often learns to produce images that score well but look wrong: over-saturated, covered in high-frequency grain, or "fried." This failure is called reward hacking. We propose to treat this failure the way quantum hardware engineers treat leakage, where a qubit's state escapes the two levels used for computation into extra levels that the error-correcting logic cannot see. In both settings the useful moves are the same. First, measure how much of the state has left the allowed subspace. Then remove that part before it builds up.

In the generative setting, the "allowed subspace" at any sample is the set of local directions that stay on the data manifold, meaning the thin surface of realistic images. The "leaked" part of a fine-tuning update is whatever pushes the output perpendicular to that surface. The key enabler is trajectory straightening. In a flow model, a generator that moves noise to data along a learned velocity field, straightening makes the path from noise to image close to a straight line. On such a path, a change to the velocity at any intermediate time maps almost linearly onto a change in the final image. That makes "the perpendicular part of the update" something we can compute exactly and cheaply. We will build a per-sample leakage meter from the pretrained model's own denoiser and a fine-tuning rule that discards the perpendicular part of each reward gradient. We will then test whether leakage predicts and causes reward hacking in offline preference optimization and online reinforcement fine-tuning.

Core Claim

Our claim is that most artifact-style reward hacking in flow fine-tuning is caused by the component of the learned velocity change that is normal to the data manifold. On near-straight flows this component can be isolated in the model's output space and projected away. Doing so recovers reward gains with fewer artifacts than penalizing all deviation from the reference model.

Write $v_\theta(x_t,t)$ for the fine-tuned velocity at noisy state $x_t$ and time $t\in[0,1]$, with $t=1$ at the data end, and $v_{\text{ref}}$ for the pretrained velocity. For a straight flow the one-step data estimate is

$$\hat{x}_1 = x_t + (1-t)\,v(x_t,t), \qquad \delta \hat{x}_1 = (1-t)\,\delta v,$$

where $\delta v = v_\theta - v_{\text{ref}}$. Let $P$ be the projector onto the estimated tangent space of the data manifold at $\hat{x}_1$. We define leakage as the fraction $\Lambda = \|(I-P)\,\delta v\|^2 / \|\delta v\|^2$. Our fine-tuning rule replaces the loss gradient with respect to the predicted velocity, $g = \partial \mathcal{L}/\partial v$, by its tangent part $Pg$ before backpropagating into the weights. The linear relation above is exact only when trajectories are straight. That gives a sharp prediction: projection should help more on straighter models.

Relation to Prior Work

Manifold Drift in Flow Preference Optimization motivates this proposal. It shows that FlowDPO gradients have a nonzero component normal to the data manifold, particularly the repulsive term on the dispreferred sample. Its fix, ThermoDPO, adds a soft winner-side anchor toward the pretrained model and is evaluated only offline. We keep its diagnosis but change the remedy in two ways. We remove the normal component instead of penalizing all drift, and we test online rollouts, which that paper leaves open.

ConsistentRFT attacks hallucinations in online fine-tuning of FLUX.1-dev. It schedules rollout noise and regularizes the deterministic velocity toward the pretrained trajectory. Like the anchor, that regularizer is direction-agnostic. It also supplies our main online baseline and its VH-Evaluator.

Blocking the Leakage supplies the method template. It tracks a protected low-rank subspace using the Marchenko–Pastur edge, a random-matrix rule for separating signal eigenvalues from noise, and projects test-time-adaptation gradients in weight space for classifiers. We reuse its spectral rank selection. However, we project in the generator's output space, using a geometric object (the image manifold's tangent) rather than a gradient-history subspace, and we apply it to generative fine-tuning.

Coupler-Assisted Leakage Reduction contributes the design principle and no technique. Leaked population there persists across cycles and must be actively flushed. Its reported 0.58% induced error on the computational subspace is a reminder that flushing has a cost, which we measure here as lost on-manifold reward progress.

Inference-time work constrains the design. MIRA shows that regularizing only the initial noise fails because the reverse process amplifies small deviations. Gradient Guidance for Diffusion Models shows that naive guidance gradients break latent structure, and it preserves that structure through a forward-prediction loss. Both support acting on the data-end estimate rather than on noise. Neither handles parameter fine-tuning of straight flows.

Research Program

The first component is a tangent estimator. For a trained flow, the posterior-mean denoiser near the data end approximately maps points onto the data support. Its Jacobian should therefore have a few large singular values, spanning the tangent space, and a bulk of small ones. For each training sample we compute a randomized truncated SVD of the pretrained denoiser Jacobian at $\hat{x}_1$, using a fixed budget of Jacobian–vector products in the latent space. We choose the rank with the Marchenko–Pastur edge criterion borrowed from MGP.

We validate this estimator on the Toy 3D Manifold Benchmark from Manifold Drift, where the true tangent plane is known. That benchmark also gives us an oracle-projection upper bound.

The second component is the leakage meter $\Lambda$. We log it along training runs of vanilla FlowDPO, ThermoDPO and online group-relative reinforcement fine-tuning, on Stable Diffusion 3.5-Medium and FLUX.1-dev. We ask whether rising $\Lambda$ predicts the later divergence between the training reward and held-out quality measures, and whether it does so better than plain $\|\delta v\|$ or KL to the reference model.

The third component is tangent-projected fine-tuning. The rule $g \mapsto Pg$ applies identically to the DPO velocity residuals and to the policy-gradient signal on the mean velocity of stochastic rollouts. We also test whether it works on the straightening axis. We take a base rectified-flow model and a reflowed, straighter version of it, measure their trajectory curvature, and check whether the projection's benefit tracks straightness as the linear map predicts. This answers a narrow version of the open question of whether straightness and resistance to hacking interact.

We are explicit about scope. Some hacks stay on the manifold, such as drawing digits instead of objects, reported in Training Diffusion Models with Reinforcement Learning, or shrinking objects, reported in Carve3D. Projection cannot stop these. We report them separately through GenEval counting and object-size statistics, so we do not overclaim.

Baselines

The naive reference is unconstrained fine-tuning. Offline this is vanilla FlowDPO; online it is reinforcement fine-tuning with KL regularization in the style of DPOK. Both use the same reward, prompts and step budget. They establish how much hacking occurs.

The strongest offline competitor is ThermoDPO-weighted, because it targets exactly the failure we target on the same OCR task and model. It reports an OCR-accuracy gain of 47.5% and a 16.0% improvement in the average of four key metrics at CFG 4.5 on SD3.5-M. On the toy benchmark it reports a StrictScore of 0.899, against 0.629 for FlowDPO and 0.857 for FlowDPO with rejection-sampling fine-tuning. We run it from the same initialization, prompts and preference pairs. Its anchor weight and temperature get the same hyperparameter-search budget as our rank threshold. Beating it means a better Pareto frontier of OCR accuracy against held-out artifact measures (VH-Evaluator, HPSv2, GenEval), not just a higher OCR score at one setting.

The strongest online competitor is ConsistentRFT on FLUX.1-dev. It reports 49% and 38% average reductions in low-level and high-level hallucinations relative to standard reinforcement fine-tuning, and a 5.1% out-of-domain GenEval gain versus the base model's −0.4%. We match its rollout count and its wall-clock budget. Our Jacobian computations are charged against that budget, so we run fewer rollouts if needed. Beating it means lower VH-Evaluator scores at equal or higher training reward. We also test the combination with its rollout schedule, since the two methods act on different parts of training.

Three ablations isolate the key ingredient. The first projects onto a random subspace of the same rank. The second keeps only the normal component. The third is weight-space MGP applied as-is to the fine-tuning gradients. If the random-subspace ablation matches the full method, the geometry is not doing the work.

Why not just…?

Why not just strengthen the KL penalty or the ThermoDPO anchor? Both penalize tangent and normal deviation equally. They therefore trade reward against artifacts along a single knob, while projection should let tangent progress through at full rate. This is a real threat: a well-tuned anchor may sit on the same frontier. Our tuning-matched frontier comparison settles it. If the anchor matches, the surviving contribution is the leakage meter as a diagnostic.

Why not just restrict updates to low-rank adapters or use weight-space MGP? Low rank in weight space does not imply staying on the manifold in image space. A single direction in weight space can still move outputs off the manifold. Blocking the Leakage validated its subspace only for classifiers and assumed spectral dominance that is untested for velocity fields. We include it directly as an ablation.

Why not just penalize high-frequency content or filter rollouts? ConsistentRFT traces hallucinations to distortion of the underlying velocity field, not only to pixel noise. Saturation-type drift reported for Pref-GRPO and DPOK is also not high-frequency. We will still run a Laplacian-variance penalty as a baseline. Because VH-Evaluator partly measures Laplacian variance, we report its vision-language-model half separately to avoid rewarding the baseline for gaming the metric.

Why not just use reward-conditioned pretraining or best-of-N? MIRO requires re-annotating the pretraining corpus and inherits the reward's biases. Best-of-N multiplies sampling cost by N, which erases the speed advantage of straight flows. We report best-of-N at matched inference compute as a reference point only.

Decisive Checks

On the toy benchmark, the estimated tangent should align with the true tangent, measured by principal angles. Estimated projection should come within a small margin of oracle projection on StrictScore.

On SD3.5-M with the OCR preference set, three outcomes would support the claim. The rise in $\Lambda$ should precede the drop in held-out HPSv2 and GenEval during vanilla FlowDPO by a measurable number of steps. Projected FlowDPO should dominate ThermoDPO-weighted on the reward-versus-artifact frontier. The random-subspace ablation should fail.

Online on FLUX.1-dev, with OCR and HPSv2 rewards, projection should reduce VH-Evaluator scores relative to ConsistentRFT at matched compute.

On the straightening axis, the reflowed model should show both lower baseline leakage and a larger benefit from projection. If the benefit is flat across straightness, the linear-map argument is wrong and the method reduces to a heuristic.

We will release logged $\Lambda$ curves alongside code built on rethinking-diffusion-rl, so that others can check whether leakage tracks hacking in their own runs.