Robot policies trained by imitation often succeed for the wrong reasons. They key their actions on the camera angle, the table texture, or which lab the demonstration came from, rather than on the object they are supposed to move. Synthetic data, such as re-rendered scenes or generated videos, can break these accidental correlations. Producing it is expensive, though: one video-world-model pipeline used 1,500 GPUs for 54 hours to make 240,000 trajectories. Most of that compute is spent uniformly, on variations the policy never relied on in the first place. We propose to first measure which shortcuts a policy is actually using and then spend generation compute only on samples that break those shortcuts. We repeat this measure-then-generate loop because removing one shortcut tends to expose the next easiest one.
The claim worth testing is quantitative. Out-of-distribution success should scale with how much each synthetic sample weakens the policy's current shortcut, not with raw sample count. If that holds, targeted synthesis should reach the out-of-distribution success of uniform randomization at a fraction of the GPU-hours. It also gives a way to read robot data scaling curves: flat curves come from data whose added volume does not touch the correlations the policy is exploiting.
Core Claim
Neural networks trained by gradient descent tend to fit the simplest pattern that lowers the loss first. This tendency is called simplicity bias. Once an easy, strong feature explains the labels, the gradient signal for harder causal features shrinks, which Gradient Starvation formalized. For a robot policy this has a practical consequence. At any point in training there is a small set of nuisance factors, such as viewpoint, background, lighting, distractor layout, or data source, that the policy leans on hardest. Synthetic samples that vary those factors while holding the task fixed carry most of the useful information. Samples that vary anything else carry almost none.
We make this operational with a cheap counterfactual probe. Take held-out frames and re-render each one with a single factor changed: a new texture, a shifted camera, or a swapped distractor. Then measure how far the policy's predicted action moves. Compare that shift to how far the action moves under a task-relevant change, such as moving the target object. The ratio is a per-factor reliance score. The generation budget for the next round goes to the factors with the highest scores, using renderings that keep the factor independent of task identity. One factor is always probed: whether a frame is real or synthetic. The reason is that targeted synthesis concentrated on some tasks could make render style itself predictive of the task, which would create a new shortcut.
The scaling hypothesis is that out-of-distribution success $S$ follows
$$S(C) \approx S_\infty - a\, C_{\text{eff}}^{-b},$$
where $C$ is generation compute in GPU-hours and $C_{\text{eff}}$ is the cumulative reduction in measured reliance on nuisance factors that this compute bought. The constants $a$, $b > 0$ and the ceiling $S_\infty$ are fitted. The power-law form is borrowed from the empirical robot scaling results discussed below and is an assumption to be tested, not a derivation. The falsifiable prediction is this: when success is plotted against $C_{\text{eff}}$, the curves for targeted, uniform, and random synthesis should collapse onto one line. When the same success is plotted against raw $C$, they should separate.
Where This Sits
Three papers motivate the gap.
Shortcut Learning in Generalist Robot Policies shows that policies trained on fragmented data, from Diffusion Policy up to $\pi_0$, latch onto viewpoint and background cues. It also shows that hand-designed viewpoint and object augmentations help, and that uncontrolled diversity can make fragmentation worse. However, the factors were chosen by hand, the augmentation was one-shot, and generation cost was never measured.
An Information-Theoretic Criterion for Efficient Data Synthesis argues that synthetic data helps only when it is anchored to signals outside the generator. It also argues that diverse coverage of task-relevant partitions beats redundant volume. That paper works on text and theorem proving and does not cover continuous control. Our physical re-rendering acts as the external anchor, and the reliance probe decides which partitions still need coverage.
DreamGen shows that generated videos labeled with inferred actions can teach new behaviors. It also documents the cost of this approach, and it never checks whether the generator's own visual artifacts become shortcuts.
Two papers constrain the design.
Real2Render2Real generates 1,000 demonstrations in 16 to 38 minutes on one GPU by replacing physics simulation with kinematic rendering. That makes it the obvious cheap generator, but it cannot be trusted on contact-heavy tasks, so we restrict real-robot claims to tasks where kinematics is enough.
LIBERO found that naive multi-task pretraining can hurt later transfer. This warns that "more data" is not a neutral baseline.
The probe-and-retrain loop borrows its intuition from Chen et al., who found that suppressing easy features early in training yields more robust models, and from Tsoy et al., who found that reliance on simple features grows as training loss approaches zero. Together these suggest that a policy's shortcuts change during training. A one-shot augmentation can therefore be aimed at a stale target.
Research Program
We first build factor-controllable generators in three settings: LIBERO-Spatial, SIMPLER (a simulator that replicates the Bridge and RT-1 robot data distributions), and RoboCasa. All three allow camera, texture, lighting, and distractor parameters to be re-rendered while replaying the same action sequence. For real-robot experiments we add a Real2Render2Real-style pipeline. Every generated sample is logged with its GPU-seconds, so compute becomes the x-axis of every plot.
Second, we validate the reliance probe. On LIBERO-Spatial and SIMPLER we deliberately plant correlations, for example by tying a table texture to each task. We then check two things: whether the probe ranks the planted factor highest, and whether its score correlates with the drop in success under that factor's shift. The probe costs only forward passes, so we report its overhead separately; we expect it to be small next to generation.
Third, we run the loop for Diffusion Policy and a small vision-language-action model (a policy that reads images and an instruction and outputs actions). Each round trains or fine-tunes the policy, probes it, generates a fixed GPU-hour tranche aimed at the top factors, and continues. We record which factor dominates each round, to test the "next easiest shortcut" prediction. Finally, we fit the scaling relation above on raw and effective compute.
Baselines
The naive reference point is training on the original demonstrations with standard crop and color jitter and no synthetic data. This sets the floor that every synthesis strategy must clear per GPU-hour.
The strongest existing method is the hand-designed viewpoint and object augmentation from Shortcut Learning in Generalist Robot Policies. On the physical AgileX PIPER arm, the paper reports that viewpoint augmentation "reduced shortcut learning by 75% and improved OOD success by 175%," and that a bridging object raised out-of-distribution success from 0.2 to 0.75. The file also quotes its SIMPLER object-augmentation result as reducing shortcut learning "from 1.0 to 0.8" with out-of-distribution success going "from 0.68 to 0.25." That pair is internally inconsistent, so we will re-run it rather than rely on it. We reproduce this baseline in simulation using the same renderer, the same policy, and the same GPU-hour budget as our method, with its augmentations fixed in advance. Beating it means higher out-of-distribution success at equal compute, or equal success at no more than half the compute, averaged over at least three seeds.
The uniform-randomization baseline spends the identical budget varying all factors independently of task. This is the real competitor, because if task-independent randomization is enough, targeting adds nothing.
On RoboCasa we also include MimicGen-style augmentation as reported by RoboCasa. The paper reports that adding 3,000 synthetic demonstrations to 50 human ones raised atomic-task success from 28.8% to 47.6%.
Three ablations isolate the key ingredient. In the first, factors are chosen at random instead of by the probe, which isolates the probe. In the second, an oracle that knows the planted factor chooses them, which sets an upper bound. In the third, the probe runs once and is not repeated, which isolates the loop. If one-shot probing matches iterative probing, the "next shortcut" story is wrong.
Why not just…?
Why not just randomize every factor uniformly? This is the alternative most likely to match us, and we treat it as the main hypothesis to beat rather than a strawman. Task-independent randomization does decorrelate nuisance factors from actions. Its weakness is that the budget is divided across every factor, including ones the policy ignores. Our prediction is that targeted synthesis wins only when there are many candidate factors and few are exploited. We will vary the number of rendered factors from three to more than ten. If uniform randomization matches targeting across that range, we will report that the gain comes from decorrelation itself, not from allocation, and the effective-compute scaling analysis becomes the main contribution.
Why not just add a training-time regularizer instead of data? Spectral Decoupling, which penalizes the size of the model's raw outputs, raised Colored MNIST accuracy from 23.7% to 68.4% in Gradient Starvation. Fourier input features improved RoboCasa success by 13 to 34% in Fourier Features Let Agents Learn High Precision Policies with Imitation Learning. Neither costs generation compute. However, Spectral Decoupling was designed for cross-entropy classification, not for diffusion action heads, and it needs careful tuning. Fourier features address low-frequency bias in coordinates, not background shortcuts. We include Spectral Decoupling adapted to the action head as a zero-compute baseline and also stack it with our method. If the regularizer alone matches us, data targeting is unnecessary.
Why not just scale model size and real demonstrations? The shortcut paper found that shortcuts persist in $\pi_0$, and LIBERO found that pretraining on more tasks can hurt transfer. Both suggest that scale alone does not remove fragmentation-driven shortcuts. We test this directly by reporting out-of-distribution success for doubled real data at its teleoperation cost alongside our compute-matched curves.
Why not just use a video world model or test-time verification? DreamGen's synthetic-only policy scored 20.6% on RoboCasa at very high cost, and RoboMonkey's 7B verifier runs at 1.5 Hz on an H100 GPU. We do not rerun DreamGen at scale. Instead we probe a small sample of its style of output for real-versus-synthetic reliance, which tests whether expensive generators add shortcuts of their own.
Decisive Experiments
The central check is the curve collapse. On LIBERO-Spatial, SIMPLER, and RoboCasa we plot out-of-distribution success under held-out viewpoints, textures, and distractors against raw GPU-hours and against $C_{\text{eff}}$, for the targeted, uniform, random-factor, and hand-designed strategies. Collapse on $C_{\text{eff}}$ together with separation on raw compute confirms the claim. No collapse refutes it.
The second check is the planted-shortcut test. We ask whether the probe recovers known correlations, and whether removing one correlation produces a measurable rise in reliance on the next.
The third check targets the source shortcut. We train with targeted synthesis concentrated on half the tasks and test whether render style becomes predictive. If it does, the mixing rule must be corrected.
Finally, on a single real arm, we repeat the strongest simulation comparison using Real2Render2Real-generated data on pick-and-place tasks that do not depend on contact physics. Success is reported per teleoperation-hour and per GPU-hour.
References
- Gradient Starvation: A Learning Proclivity in Neural Networks
- Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
- An Information-Theoretic Criterion for Efficient Data Synthesis
- DreamGen: Unlocking Generalization in Robot Learning through Video World Models
- Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
- Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
- Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data
- RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots
- Fourier Features Let Agents Learn High Precision Policies with Imitation Learning