Learned Multi-View Constraints That Do Not Fade: Bounding Reward Hacking During Exploration

Hand-written rewards for robots routinely leave loopholes. A humanoid rewarded for getting close to a chair learns to perch on the chair leg or lean at an odd angle. A walker rewarded for forward speed learns a gait that slams its joints. Recent work closes some of these loopholes by scoring the agent from several camera angles with a video-language model and adding that score to the reward. This works for teaching the task. It does not stop the agent from breaking the rule while it is still learning, and by design the visual signal switches off once the agent becomes competent. We propose to use the same multi-view judgment for a different job: as a constraint, a separately budgeted cost that the learner must keep low for the whole of training and deployment, rather than one more reward term that can be traded away or that decays.

The core idea is short. Ask a judge, looking at synchronized videos of a rollout from several cameras, which behaviors the hand-written reward fails to rule out: unnatural contact, self-collision, tipping, joint abuse. Distill those judgments into a cheap learned cost on the robot's internal state. Then optimize the task reward subject to a hard budget on that cost with standard constrained reinforcement learning. The question is whether a learned, view-invariant cost can stand in for the hand-engineered safety terms that constrained RL currently assumes exist. Specifically, does it keep cumulative violations low during exploration without collapsing into a motionless policy?

Core Claim

Multi-view visual feedback is more useful as a persistent constraint than as decaying reward shaping, for one class of failures: failures that are easy to recognize on video but hard to write as equations. Formally, with task reward $r$ (hand-written, possibly loophole-ridden), optional decaying visual shaping $b_t$, and a learned multi-view cost $c_\phi$ with parameters $\phi$, we solve

$$\max_\pi \; \mathbb{E}\Big[\sum_t \gamma^t \big(r(s_t,a_t) + b_t(s_t)\big)\Big] \quad \text{s.t.} \quad \mathbb{E}\Big[\sum_t \gamma^t c_\phi(s_t)\Big] \le \epsilon,$$

where $\epsilon$ is a cost budget and $b_t \to 0$ as the agent matches a reference set of good trajectories. $c_\phi$ never decays. We claim three things. First, this separation removes loopholes that reward shaping alone leaves behind once shaping hands over. Second, it gives fewer violations during training than adding the same visual score as a fixed penalty. Third, the multi-view cost beats a single-view cost of the same size, because failures such as a hidden limb in self-contact are occluded from any one camera.

Where This Sits

The closest work is MVR: Multi-view Video Reward Shaping for Reinforcement Learning. MVR fits a Bradley-Terry state-relevance function (a model trained from pairwise "which is better" judgments) from multi-view ViCLIP video scores. It adds the resulting reward with an automatic handover that decays to zero, and it shows on HumanoidBench "Sit_Hard" that this penalizes chair-leg sitting. We reuse its method for turning expensive video judgments into a cheap state function. We differ in where the signal enters the problem: MVR's signal is reward that disappears, ours is a constraint that persists, and MVR places no bound on unsafe exploration.

SafeDreamer solves vision-based constrained problems with world models. It uses single-camera input and benchmark-supplied, hand-specified costs, so it presupposes exactly the cost function we propose to learn. DROPJ is the nearest attempt to learn safety from feedback: human preferences with categorical justifications inside a learned world model. It enforces safety only by reweighting a reward loss and uses a single view. We borrow its idea of justification categories as the label vocabulary for $c_\phi$, but we enforce the result as a budgeted constraint. CoRe automates reward writing with LLM code plus a VLM-judged residual. It is single-camera and explores without any cost bound; it motivates our choice to leave the task reward hand-written or code-generated and learn only the constraint.

Several papers constrain the design. MetaDrive documents that folding safety into reward as a fixed penalty is brittle: agents are either reckless or "safe but useless." This is why we do not simply add $c_\phi$ to $r$. Safety Gymnasium shows plain Lagrangian updates oscillate and that PID-damped multipliers (CPPO-PID) fix this, so we use PID Lagrangian updates from its SafePO library. Breaking Safety Paradox with Feasible Dual Policy Iteration warns that a safer agent produces fewer violating samples, which starves a learned cost of negatives. Our design has to confront this directly, and it is the main technical risk. From 3D generation, MVReward and Eval3D motivate the multi-view premise: per-view 2D scoring misses cross-view inconsistency (Eval3D's multi-view geometric probe agrees with experts 83% of the time versus 46.9% for GPT-4V). We treat that as evidence that joint multi-view judgment catches failures single views miss, not as a method we import.

Research Program

The system has three parts.

The first is a labeling pipeline. We render each rollout segment from four to six fixed cameras around the robot. A VLM judge receives the synchronized clips together with a short natural-language list of prohibited behaviors for the task, such as "no body part other than feet and hips touches the chair" or "torso must not tip past horizontal." The judge returns pairwise comparisons with a categorical justification in the DROPJ style. A small human-labeled audit set, a few hundred clips per task, measures judge accuracy and is never used for training.

The second part is cost distillation. Following MVR, we fit $c_\phi$ on proprioceptive state, not pixels, from these comparisons, with an auxiliary loss aligning state embeddings to the multi-view video embeddings. The robot's internal state includes joint angles, velocities and contacts. Working on this state means the policy loop never calls the VLM at run time; the VLM is queried only on a subsample of segments. This addresses the latency and API cost that the file flags for CoRe.

The third part handles the safety paradox, which gets the most attention. We keep a persistent buffer of every segment ever flagged as violating, so negatives are never forgotten as the policy gets safer. We also spend a fixed fraction of the labeling budget on segments where $c_\phi$ is near the decision boundary or where an ensemble of cost heads disagrees. We will measure directly whether $c_\phi$'s calibration on the audit set degrades over training. If it does, the paradox is biting and the buffer and boundary queries are not enough.

Two design questions are settled empirically. The first is how early in training $c_\phi$ must be trustworthy. We bootstrap it from rollouts of a random or briefly trained policy before constrained training begins, since violations are plentiful early. The second is whether the constraint should act on episode cost, as in the display equation above, or on a per-state cost-so-far bound in the style of Backward Value Functions. The main line uses PID Lagrangian PPO because it is the most robust off-the-shelf choice. BVF is a secondary variant for tasks where training-time violations matter most.

Baselines

The naive reference point is the hand-written task reward alone, trained with the same PPO backbone. It shows how much hacking the reward permits.

The strongest existing method on the loophole question is MVR as published. That means multi-view ViCLIP shaping with decay, no constraint, run with the authors' reward construction and the same camera rig, labeling budget and environment steps. On HumanoidBench, MVR reports the best average rank of 1.67 versus TQC 3.11, RoboCLIP 2.89 and VLM-RM 3.78, but those ranks measure task return. We found no reported result on Lacuna for violation rates or unnatural-behavior frequency under MVR, so we will measure them ourselves.

To beat MVR we need three things at once. We must match its task success within a small tolerance, with success judged by the human audit, not the hackable reward. We must significantly reduce the final-policy rate of audited prohibited behaviors. And we must reduce cumulative violations during training.

The strongest competitor on the safety side is PID Lagrangian PPO (CPPO-PID from SafePO) given a hand-engineered cost. This cost is written by us, with a fixed engineering effort, from simulator contact and tilt signals that approximate the same prohibited-behavior list. It is the "oracle-ish manual engineering" competitor. Safety Gymnasium reports that PPO-Lag reduces violations by 98% relative to PPO with a 45% reward drop on velocity tasks, which frames what a good constrained learner achieves when the cost is known. Our method succeeds if it approaches this baseline's violation levels without anyone writing the cost. It succeeds more strongly if it beats the baseline on behaviors the hand cost failed to capture, measured by the human audit.

Three ablations isolate the key ingredients, each using identical labels and budgets. The first uses the multi-view judge's score as a fixed-weight penalty folded into reward, swept over five weights with the best chosen on validation; this tests "constraint versus penalty." The second applies MVR-style decay to the cost; this tests "persistent versus decaying." The third trains $c_\phi$ from a single camera judged alone, with the same number of labels, choosing the best single camera in hindsight; this tests "multi-view versus single-view." All methods get the same environment steps, the same number of VLM queries, and the same hyperparameter search budget of twenty trials per method. We use five seeds per task.

Why not just…?

Why not just keep MVR's visual reward switched on instead of decaying it? MVR introduced the decay specifically because persistent visual reward causes the policy to chase visually attractive poses at the expense of the task. A never-decaying reward therefore trades task success for appearance, whereas a constraint only activates when the budget is exceeded. This is testable: we include "MVR without decay" as an extra arm, and if it matches our method on both success and violations, the constraint formulation adds nothing and we will say so.

Why not just write the cost by hand from simulator contacts and joint angles? For self-collision or joint-limit hits this is genuinely easy, and it is a real threat. That is why the hand-cost CPPO-PID baseline is included. The case for learning rests on the behaviors in our prohibited lists that are awkward to formalize, such as "sits naturally" or "doesn't lean unnaturally," which MVR's Sit_Hard study shows a distance-based reward misses. We will report results separately for "formalizable" and "hard-to-formalize" prohibitions. We expect the hand cost to win or tie on the first group; our claim stands or falls on the second.

Why not just add the visual penalty to the reward with a tuned weight? MetaDrive documents that fixed penalties oscillate between reckless and frozen policies, and SafeDreamer reports episode cost 14.1 for PPO with penalty shaping versus 2.2 for its constrained formulation on MetaDrive. Our penalty-sweep ablation checks whether the same holds for a learned visual cost. It could plausibly fail to: a learned cost is smoother than a crash indicator. If a well-tuned penalty matches the constrained version, the practical recommendation becomes the simpler method.

Why not just use one good camera? MVR attributes part of its gain to multi-view invariance under occlusion. However, the best single camera chosen in hindsight may suffice for most prohibitions. The single-view ablation with matched labels settles this, and we will break down its errors by whether the violating body part was occluded in that view.

Proposed Experiments

The primary testbed is HumanoidBench, using the posture and locomotion tasks MVR used (Sit_Hard, Stand, Walk, Run, Slide), since that is where hand rewards visibly hack. The secondary testbed is Safety-Gymnasium's Ant and Doggo navigation and velocity tasks. There, benchmark-supplied costs let us test a sanity condition: when the prohibited list is "exceed the velocity limit" or "enter a hazard," does a learned $c_\phi$ recover performance near CPPO-PID with the true cost? A MetaWorld subset tests transfer to manipulation, with prohibitions such as "don't strike the object with the forearm."

We report four metrics:

  • task success judged by the human audit;
  • final-policy prohibited-behavior rate on the audit set;
  • cumulative prohibited-behavior count during training, computed post hoc by the audit-trained judge on logged rollouts;
  • calibration of $c_\phi$ over training on the audit set.

Hidden simulator metrics give a label-independent cross-check. These are non-foot contact forces, self-collision counts and joint-limit hits, which are never shown to the learner.

The decisive checks are three.

On hard-to-formalize prohibitions, the constrained learned cost must beat MVR on prohibited-behavior rate without losing audited success. Otherwise the core claim fails.

On Safety-Gymnasium tasks with known costs, the learned cost must come within a modest margin of the true-cost CPPO-PID violation rate. Otherwise the learned cost is too noisy to act as a constraint.

The calibration curve of $c_\phi$ must not collapse late in training. If it does, the safety paradox has defeated the buffer-and-boundary-query design, and the next step would be the world-model imagination route of DROPJ and SafeDreamer to generate violations without incurring them.