Robots are increasingly trained with reinforcement learning (RL, trial-and-error learning that maximizes a reward score) where the reward comes from a vision-language model (VLM), a model that reads images and text. The VLM watches a video of the robot and grades how far the task got. This removes the need to hand-write a reward formula for every task. It also has a blind spot. A camera cannot see force. A policy that rams a peg into a hole, crushes a handle, or jams a drawer at full stiffness can look exactly like one that does the job gently. We expect the cheapest way to raise a purely visual progress score in contact-rich tasks to be to become rigid and push hard. If so, VLM rewards will quietly teach robots destructive behavior. This is a physical version of the presence bias that Reward Hacking in Rubric-Based Reinforcement Learning documented in text rubrics, where 90.2% of rubric weight rewarded visible features and only 8.6% penalized errors or side effects.
This note proposes to measure that failure under controlled conditions and to test a specific remedy. The remedy keeps the VLM as the judge but shows it the measured wrench, meaning the force and torque at the wrist, rendered as a plot next to the video. The judge then answers two questions: an ordinary progress question, and an absence question such as "did the robot avoid excessive force for this object?" The bet is that the VLM's semantic knowledge of objects (an egg versus a steel nut, a whiteboard that must be pressed versus a hole that must be entered gently) lets it set a task-appropriate force standard. A single hand-set force threshold cannot do that. The policy acts in a variable-impedance action space, choosing at each step both a target pose and how stiff or soft to be. This makes the tradeoff between stiffness and compliance the variable through which hacking either happens or is prevented.
Core Claim
Our claim has two parts, and both can be falsified. The first is that RL against a vision-only VLM progress reward, in a variable-impedance action space, will raise the learned stiffness during contact and push peak contact forces past object-damage thresholds that the judge cannot see. True task value will then fall while the proxy score rises. The second is that an absence-based force criterion, judged by the same VLM from a rendered force trace, will close most of this gap. It should match or beat a per-task hand-tuned force penalty, and clearly beat a single global threshold, on tasks whose acceptable force differs by object and phase. We do not claim the result is unhackable. Defining and Characterizing Reward Hacking proves that any non-trivial simplification of the true reward is hackable over a rich enough policy space. Our aim is to shrink the gap measurably and to find out where it moves next.
Relation to Prior Work
Three works motivate the gap. RoboReward trains Qwen3-VL judges to score robot videos on a 1–5 progress rubric and uses counterfactual near-miss negatives against gaming. It is the strongest visual reward model in the file, but its rubric is entirely about what is visible, and its corpora carry no force data. CoRe combines LLM-written reward code with VLM video preferences and reports 99.0% Meta-World success. It never parameterizes or grades impedance, and it judges contact only from pixels. CompliantVLA-adaptor is the closest work. It prompts a VLM about once per second to choose stiffness and damping by task phase, with a hard 30 N reactive cutoff underneath. There the VLM is a controller, not a reward, and no policy is trained by RL against it. Its finding that vision-language-action policies often violate 30 N supports our premise that vision-only systems are force-blind.
Three works supply the method. VICES gives the action space: the policy outputs a pose plus stiffness and damping gains at 20 Hz, tracked by a 500 Hz inner loop. Its agents learned to soften during contact only because a hand-written energy penalty told them to. IMP-HRL shows impedance-aware RL reaching 86% on Robosuite Wipe against 42% for a rigid baseline, but it relies on hand-designed affordance rewards and hand-tuned gain parameters. Our proposal replaces exactly those hand-written terms with a VLM-judged absence criterion. UMI-FT shows that cheap fingertip force sensing changes outcomes: 92% wiping success against 16% for vision-only Diffusion Policy. That result tells us force must enter somewhere. We put it into the reward rather than the policy input.
Two works constrain the design. The Skalse et al. theorem says not to expect zero hacking, so the evaluation must track a held-out true objective throughout training. The rubric-hacking paper warns that weak verifiers get exploited more over time: its exploitation rate climbed from 39% to 65%. So we test whether the policy learns to fool the judge's reading of force plots, not just whether early numbers look good.
Research Program
We first build a fragile-contact variant of Robosuite (robosuite) on Door Opening, Surface Wiping, Block Lifting, and Nut Assembly, plus a peg-insertion task. Each object receives a hidden, object-specific damage model. When peak or integrated contact force exceeds that object's threshold, the episode is marked damaged, with no visible change in the rendering. Thresholds vary deliberately across tasks. Wiping requires a minimum sustained normal force to count as cleaned. Insertion and fragile lifts tolerate only low force. Door handles fall in between. The true reward is task success without damage, plus wiping coverage where it applies. No judge or reward ever reads it.
Policies are trained with Soft Actor-Critic in the VICES action space under several reward sources, which are described in the Baselines section. Reward latency is not a real obstacle here. The judge scores short video segments offline after each rollout chunk, about one query per second of robot time, so it never sits inside the 500 Hz loop. For each run we log, over the course of training, the proxy score, the true reward, damage rate, peak and mean contact force, and the learned stiffness broken down by task phase (free motion, approach, contact). Phases are labeled from simulator contact state.
The proposed judge is Qwen3-VL-8B. We use the released RoboReward checkpoint if it is available and fine-tune it; otherwise we start from the base model. The judge receives the RGB clip and a rendered strip plot of wrist wrench magnitude and normal force over the same window, with the time axis aligned to the clip. Its output has two parts: the 1–5 progress grade, and a 1–5 absence grade answering "contact force was appropriate for this object and step, without excessive pressing or jamming." The reward is the product of the normalized progress and absence grades. We use a product so that progress cannot buy back damage. Before any RL, we calibrate the absence grade on a small labeled set of segments from scripted rigid and compliant controllers, sized like RoboReward's few-hundred-example calibration regime. This mirrors how Robot Fine-Tuning Made Easy found that small target-domain sets fix VLM detector errors. We report the judge's accuracy at reading force plots separately from its task judgment, so that the two failure modes can be told apart.
Baselines
The naive reference point is RL against the vision-only progress judge: the same Qwen3-VL-8B judge and the same RoboReward rubric, with no force information. This is the condition we predict will hack, and every other run shares its data, judge size, number of environment steps, and seeds. The relevant reported figure for this judge class is RoboReward 8B's mean absolute error of 0.665 on RoboRewardBench, and its 80% drawer-opening success on a WidowX arm against 90% for human rewards (RoboReward). These figures say nothing about force, and no reported result found on Lacuna measures contact force under RoboReward-trained policies. That measurement is our first deliverable.
The strongest hand-engineered competitor is the vision-only judge plus a per-task hand-tuned force penalty. For each task, an engineer tunes a quadratic penalty above a threshold, in the style of VICES's energy terms. Its tuning budget equals our calibration budget, measured in the same number of labeled episodes or sweep evaluations. The closest reported numbers for impedance RL with hand-crafted rewards are IMP-HRL's 86% versus rigid MAPLE's 42% on Robosuite Wipe (IMP-HRL). VICES reports near-100% zero-shot Panda-to-Sawyer transfer on Path Following and Door Opening but gives no numeric success values in text. Because these come from different reward setups, we rerun IMP-HRL-style rewards inside our fragile benchmark instead of borrowing its numbers.
The strongest existing safety mechanism is CompliantVLA-adaptor's design pattern: a global 30 N reactive stiffness cutoff applied on top of the vision-only reward. Its paper reports average success rising from about 9.8% to 17.3% under a strict 30 N limit on its LIBERO/ManiSkill suite (CompliantVLA-adaptor). We also include a CoRe-style baseline, in which an LLM writes a per-task force-budget reward function from the task description. This is the automated, non-visual way to make force limits depend on the task.
Two ablations isolate the key ingredient. In the first, the judge receives the force plot but only the progress question, which tests whether access to force alone helps or whether the explicit absence criterion is needed. In the second, the judge receives the absence question with a numeric force summary as text instead of a plot, which tests whether rendering force as an image matters. Beating the baselines means two things. Damage-free success, our true reward, must exceed the vision-only judge by a clear margin on the fragile tasks. It must also be at least equal to the per-task hand-tuned penalty and above the global threshold on tasks where the ideal force differs strongly from 30 N, namely wiping and fragile insertion. All of this must hold across five seeds, with confidence intervals reported.
Why not just…?
Why not just add a fixed force penalty or a 30 N cutoff to the visual reward? A single threshold is a real competitor on tasks where safe force happens to sit near it, and we expect it to do well there. It should fail in two opposite ways elsewhere. On wiping, which needs sustained pressure, it should push the policy toward contact too light to clean. On fragile insertion, it should allow damage below 30 N. The CompliantVLA-adaptor result that success reached only 17.3% under a strict 30 N limit hints that a single number is costly. Our benchmark varies thresholds across tasks precisely to test this. If the global threshold matches our method, we will report that the semantic judge adds nothing.
Why not just hand-tune a force penalty per task? This is the most serious threat, and it may well match our method on a handful of tasks. It is also the manual reward engineering we are trying to remove, and it scales poorly, as MineDojo argues for many-task settings. We give it the same tuning budget and also evaluate it on held-out object variants whose damage thresholds differ from the tuned ones. That test shows whether hand tuning transfers where a judge that reads the object should.
Why not just have an LLM write force-budget reward code, CoRe-style? Code-written budgets depend on the task but cannot see the object instance. They are also exposed to their own gaming, since CoRe needed preference alignment to suppress hacking shortcuts (CoRe). Both conditions run with matched query budgets. If LLM-written budgets do as well, the conclusion is that force-aware rewards matter and the visual judge is unnecessary, which would itself be worth reporting.
Why not just clone force-annotated demonstrations and skip RL? UMI-FT shows this works when demonstrations exist. It needed 630 in-the-wild demonstrations to stop failing in novel scenes (UMI-FT). Our question concerns the reward models now being used to fine-tune such policies, so behavior cloning is a complementary pipeline rather than a substitute.
Decisive Checks
The first check establishes that the problem exists. Under the vision-only judge, the proxy score must rise while damage-free success falls, and stiffness during contact must rise over training. If stiffness stays moderate and damage stays low, the force-blind hacking hypothesis is wrong for these tasks, and the project narrows to a negative result about RoboReward-style rewards. That result would still be useful.
The second check decides the method. On tasks whose ideal force differs strongly from 30 N, the absence-graded judge must beat both the global threshold and the LLM-written budget on damage-free success. On held-out object variants, it must at least match the per-task hand-tuned penalty.
The third check looks for where hacking moves. We track the gap between the judge's absence grade and true damage over training. We specifically look for behaviors that game the plot, such as short force spikes that the plot rendering compresses visually, and periodic contact that keeps the plotted force low while still causing damage. The same gap-tracking approach is how the rubric study measured its rise from 39% to 65% exploitation. A rising gap would trigger a variant in which a second, cross-family judge rescores the force plot.
A small hardware confirmation on a Franka arm with a wrist force sensor will cover wiping and drawer opening. It reports peak force and success for the vision-only judge versus the absence-graded judge. The rigid condition runs only under a hardware force limit, since IMP-HRL judged rigid RL too hazardous for real hardware.
References
- Reward Hacking in Rubric-Based Reinforcement Learning
- Defining and Characterizing Reward Gaming
- RoboReward: General-Purpose Vision-Language Reward Models for Robotics
- CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
- CompliantVLA-adaptor: VLM-Guided Variable Impedance Action for Safe Contact-Rich Manipulation
- Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks
- Impedance Primitive-Augmented Hierarchical Reinforcement Learning for Sequential Tasks
- In-the-Wild Compliant Manipulation with UMI-FT
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning
- MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge