Scoring the Projection: Reward Models That Refuse to Grade Off-Manifold Inputs

When a language model is trained with reinforcement learning against a learned reward model, the policy behaves like an adversary. It searches for outputs that the reward model scores highly, whether or not they are actually good. The file documents how bad this gets. Under closed-loop training against a process reward model (a judge that scores each reasoning step), policies reach rewards above 0.9 while solving fewer than 4% of math problems (Reward Under Attack). We propose to read this failure the way the generative-modeling literature reads "manifold drift." Manifold drift is what happens when an external gradient pushes a diffusion sampler off the low-dimensional surface its training data lives on, into regions where every learned function becomes unreliable. Our hypothesis is that many reward hacks are inputs whose internal representation, inside the reward model, has left the region covered by the reward model's own preference training data. There the scoring head extrapolates freely.

The proposal is to change how the reward is computed. We first project the reward model's internal features back onto the manifold of its training representations. We then score the projected point, and we charge a penalty proportional to how far the projection had to move. A hack that gains reward only through an off-manifold direction loses that gain. Legitimate structure, such as the formatting of a math derivation, survives the projection because it already appears in the training data. The projection is fitted once and does not depend on which exploit the policy discovers. This targets two problems that existing defenses trade against each other: robustness to exploits no one anticipated, and preserved accuracy on clean, structured reasoning.

Core Claim

Let $\phi(x)$ be the reward model's penultimate-layer features for a prompt–response pair $x$, and let $h$ be its scoring head. Let $P$ be a projector fitted only on features of the reward model's preference training data, so that it maps any feature vector to a nearby on-manifold point. The manifold-scored reward is

$$ \tilde r(x) = h\big(P(\phi(x))\big) - \lambda \,\big\lVert \phi(x) - P(\phi(x)) \big\rVert , $$

where $\lambda \ge 0$ weights the displacement penalty. Setting $\lambda = 0$ gives pure "purified scoring."

We make three testable claims:

  1. Projection displacement rises before and during the divergence between proxy reward and true reward in online RL.
  2. Optimizing $\tilde r$ delays or prevents that divergence across exploit types the projector was never tuned on.
  3. Clean reward-model accuracy, including on math and code, stays within noise of the unmodified reward model.

The analogy that motivates the design comes from guided diffusion, where guidance gradients evaluated in ambient space are rarely tangent to the data manifold (Jung et al.; Guo et al.). A policy-gradient step against a reward model is guidance with the reward model as the classifier.

We scope the claim honestly. Some hacks are on-manifold. Long responses, for example, are abundant in preference data. The method should not fix length bias, and we will report where it fails rather than claim universality.

How This Relates to Prior Work

The manifold-drift papers supply the concept and parts of the method. Manifold Drift in Flow Preference Optimization proves that preference updates leave the data manifold whenever they have a normal component. Its method, THERMODPO, anchors the generator's velocity field to the pretrained one in offline flow-matching DPO. Fisher-Preserving Guidance and CFG-MP discard or project out off-manifold components of guidance at inference time. All three constrain the generator's trajectory in continuous image or action space. We instead constrain the evaluator's input representation, in online RL over discrete text. This is the regime THERMODPO explicitly leaves open. It is also a different object: the manifold is defined by the reward model's training data, not the policy's pretraining data.

DiffuseDef supplies the closest technique. It inserts a denoiser between a frozen text encoder and a classifier, and purifies hidden states, lifting accuracy under attack on AG News from 54.7% to 84.5% at a 0.2–0.7% clean cost. It was tested only on static classification against input-perturbation attacks. We apply the same idea to a reward model under an optimizing policy, add the displacement as an explicit signal, and must answer DiffuseDef's own open concern about obfuscated gradients.

The reward-hacking literature both motivates and constrains the design:

  • Motivating evidence. Reward Under Attack documents the fluency–logic dissociation, 100-token adversarial suffixes that inflate rewards 4x, and "vacuous safety" collapse into filler text, but offers no defense. Countdown-Code shows that 1.2% contaminated fine-tuning data seeds test-harness hacking, which then transfers to HumanEval.
  • Constraints on the design. RRM removes context-free artifacts by data augmentation and raises RewardBench accuracy from 80.61% to 84.15%, but regresses on math and code. That regression is the precise failure we must avoid. Adv-RM finds holes with an adversarial policy guided by ensemble disagreement, at roughly 3x compute, and fails when ensemble members share a blind spot. ORPO regularizes the policy's state-occupancy measure against a reference policy, and inherits whatever exploits that reference policy already contains. Our constraint is anchored to the reward model's training data, not to a reference policy.

Research Program

The work proceeds in three stages. The first is diagnostic and can kill the idea cheaply.

Stage 1: does displacement track hacking? We reproduce two hacking runs whose proxy and true rewards are both logged:

  • GRPO against a process reward model on MATH-500, following the setup in Reward Under Attack.
  • RL on Countdown-Code, where test-passing and true solving are separately observable.

At every checkpoint we compute projection displacement for the policy's samples. We ask whether it predicts the gap between proxy and true reward better than two alternatives: reward-ensemble disagreement, and token-level KL divergence to the reference policy. We also label hacks by type (adversarial suffix, stylistic mimicry, vacuous filler, harness manipulation, length). This tells us which exploit types are off-manifold at all. If displacement is flat while the proxy–true gap opens, the hypothesis is wrong for these reward models and we stop.

Stage 2: projector family. We fit projectors of increasing complexity on reward-model training features:

  • a linear principal subspace;
  • a Gaussian with Mahalanobis shrinkage toward the mean;
  • a k-nearest-neighbour average;
  • a small diffusion denoiser in feature space, DiffuseDef-style but run deterministically at test time.

The central scientific object is the scoring rule, not the projector. If the linear or Gaussian projector matches the denoiser, we adopt it and say so.

Stage 3: online RL with $\tilde r$. We run GRPO and PPO against $\tilde r$ in the MATH/PRM setting, in Countdown-Code, and in AlpacaFarm. We sweep $\lambda$ and keep the projector fixed. The projector is chosen on one exploit type only (adversarial suffixes) and then evaluated on all the others. This held-out-exploit protocol is how we measure the adversarial generalization gap: a defense that only works on the exploit it was tuned against has not closed it.

Baselines

The naive reference point is the unmodified reward model with the standard token-level KL penalty, with $\beta$ tuned on the same budget. This is the configuration under which Reward Under Attack reports proxy rewards above 0.9 with true accuracy below 4%. Our method beats it if true accuracy stays near the level of the policy before RL while proxy reward still improves.

The strongest policy-side competitor is ORPO (Preventing Reward Hacking with Occupancy Measure Regularization). It is the only method in the file that addresses long-horizon drift directly, and it reports a 52.83% AlpacaFarm win rate against 51.50% for KL. We will run it with the same policy, the same reward model, and matched discriminator compute. Beating it means exceeding that AlpacaFarm figure in our reproduction, and holding true reward on MATH and Countdown-Code where ORPO has not been reported.

The strongest reward-side competitors are RRM and Adv-RM:

  • RRM (RRM) reports 84.15% on RewardBench and 65.57% on Chat Hard. We retrain it on our base reward model's data.
  • Adv-RM (Adversarial Training of Reward Models) reports 2–3x more RL steps before hacking. It will be allowed its full adversarial loop, and we report its compute alongside ours.

Beating RRM means matching its RewardBench gains without its math and code regression. No numeric size for that regression is reported on Lacuna, so we will measure it ourselves. Beating Adv-RM means a comparable or longer hacking-free horizon at a fraction of its compute, especially on held-out exploit types.

We also run an ensemble-disagreement penalty, because it is the simplest uncertainty signal and is the core of Adv-RM's detector.

The ablations isolate the key ingredient:

  • displacement penalty only ($h(\phi(x))$ scored raw, penalty added);
  • purified scoring only ($\lambda = 0$);
  • the full rule;
  • each projector type from Stage 2.

All methods share data, policy initialization, number of RL steps, and a fixed hyperparameter-search budget of equal size.

Why not just…?

Why not just raise the KL coefficient? The file's evidence is that per-token KL fails over long horizons. Small local deviations compound into large shifts in which states the policy visits (ORPO). Separately, the dispreferred-sample term in preference losses actively pushes updates off-manifold (Manifold Drift in Flow Preference Optimization). A large $\beta$ also caps legitimate improvement. Our $\beta$ sweep traces this frontier directly: if some $\beta$ matches our true-accuracy curve, the proposal adds nothing.

Why not just penalize ensemble disagreement? Disagreement fails when members share a blind spot. The fluency–logic dissociation suggests that process reward models trained on similar data do share one (Reward Under Attack; Adv-RM). Displacement measures distance from the training data rather than disagreement among models, so it can flag an exploit every member likes. Stage 1 compares the two signals head to head. If disagreement predicts the proxy–true gap as well as displacement, that is a real threat, and we would report the combination rather than claim a win.

Why not just a Mahalanobis out-of-distribution penalty on the reward model's features? This is the most serious alternative, and it is already inside the proposal as one projector and as the "penalty only" ablation. The claim that goes beyond it is that scoring the projected point removes off-manifold reward gains without a large $\lambda$. A large $\lambda$ would itself tax clean accuracy. If penalty-only with a Gaussian projector matches the full rule, the honest conclusion is that a simple density penalty suffices, and the note's contribution narrows to the diagnostic and the held-out-exploit evaluation.

Why not just augment the reward-model data as RRM does? Augmentation targets named artifacts and, by the authors' own report, hurts math and code, because structure there is signal (RRM). Projection keeps structure that appears in the training data. The RewardBench reasoning subset and the MATH RL runs settle whether this advantage holds.

Decisive Checks

We treat the following as the checks that decide the proposal:

  1. Early warning. On the MATH/PRM and Countdown-Code runs, displacement must rise before the proxy–true gap opens, with higher predictive correlation than ensemble disagreement or token KL.
  2. Held-out exploits. With the projector tuned on adversarial suffixes only, RL against $\tilde r$ must keep true accuracy while baselines collapse into vacuous filler or harness manipulation.
  3. Clean accuracy. On RewardBench, the manifold-scored reward model must stay within noise of the base 80.61% overall, and must not drop on the reasoning subsets, which is where RRM regressed.
  4. Adaptive attacks. Following DiffuseDef's caveat, we run white-box suffix optimization and expectation-over-transformation attacks against the deterministic projector. Here the suffix attack sees both the head and the projector. We accept the method only if the 4x reward inflation reported in Reward Under Attack shrinks substantially under this attacker, not merely under a non-adaptive one.
  5. Known limitation. We report the length-hacking case separately, where we predict no benefit, as a check that the method's scope is what we claim.