Hidden Noise Levels Cause the Clean-Precision Cost of Recovery Data in Robot Imitation

Robots that learn by imitation are often made more robust by synthesizing extra training demonstrations. A small set of human demonstrations is geometrically "warped": each motion is re-expressed relative to the object it acts on, then moved to wherever that object sits in a new scene. Recent systems add a further step. They corrupt the robot's perception of where objects are, let the policy make mistakes, and train it on warped recovery motions. These systems report large gains under noise. None reports what happens on the clean, well-calibrated setups that the robot also faces. In image classification, the analogous move of training for robustness to perturbed inputs reliably costs accuracy on unperturbed inputs.

We argue that the same cost appears in robot imitation for a specific and fixable reason. Trained on a mixture of clean and noise-corrupted observations, a policy cannot tell from a single observation how much to trust it. It therefore learns actions that hedge over where the object might really be. On clean inputs this hedging becomes a systematic positional bias, and the bias is fatal in millimeter-tolerance tasks such as nut insertion. We propose to measure this cost directly. We then remove it by giving the policy an explicit, test-time-estimable signal of its own perceptual uncertainty during training, so it can be precise when perception is good and recovery-seeking when it is not.

Core Claim

The claim has three parts. The first is that robustness-oriented warped data, in the style of Interventional MimicGen, lowers clean success on high-precision tasks. The second is that the size of the drop follows a predictable pattern: it grows with the training noise scale and with the tightness of the task tolerance. The third is that most of the drop disappears when the policy is conditioned on a perception-uncertainty estimate, while the robustness gain is preserved.

The mechanism is simple enough to state. Let $p$ be the true object pose, $\hat p = p + \epsilon$ the observed pose with noise of scale $\sigma$, and $a^*(p)$ the correct object-relative action. A policy trained on a mixture of noise levels learns roughly $\mathbb{E}[a^*(p)\mid \hat p]$ when it regresses, or a correspondingly widened action distribution when it is a diffusion model. When $\sigma$ is drawn from a mixture and never observed, the clean case $\sigma=0$ inherits this shrinkage toward the prior over poses. That produces an offset on the order of $\bar\sigma^2/(\bar\sigma^2+s^2)$ times the pose deviation, where $\bar\sigma^2$ is the average training noise variance and $s^2$ the spread of object poses; this is the standard Gaussian posterior-mean shrinkage factor. Conditioning on $\sigma$ lets the policy recover $a^*(\hat p)$ exactly when $\sigma=0$.

This gives a concrete reading of a disagreement in the robustness literature. A Closer Look at Accuracy vs. Robustness shows that image classes are well separated relative to attack radii, so an ideal classifier need not trade accuracy for robustness, and the observed loss is a generalization failure. Our setting differs in one respect: clean and noisy observations of different poses genuinely overlap. That makes the trade-off intrinsic to the unconditioned problem. Adding the uncertainty variable is what separates them again.

Research Program

The work has three components.

Recovery-data generator without human seeds. Interventional MimicGen deploys a base policy under injected pose noise and collects about ten human corrections. It then warps those corrections to new mistake states. We replace the human seeds with an autonomous procedure. When the noisy closed-loop policy drifts, we use the simulator's privileged true object pose. We interpolate from the current end-effector pose to the appropriate object-relative source segment, which is the segment-and-interpolate step of MimicGen. Each candidate is executed and kept only if it succeeds. This makes the noise scale and the volume of recovery data fully controllable, which a clean trade-off study needs. Because it departs from I-MG, we also keep human-seeded I-MG as a baseline.

Measurement of the trade-off. We train diffusion policies on nominal warped data plus recovery data at several noise scales and several mixing ratios. For each policy we record two things: success and terminal placement error on clean episodes, and success across a sweep of test-time noise. This produces a robustness-versus-clean-success frontier analogous to the $\lambda$ sweep in TRADES, which trades clean accuracy against robustness through one regularization weight. Our knob is the data mixture, not a loss term.

Uncertainty-conditioned training. Each training sample carries the noise scale under which it was generated, fed to the policy as an extra input. At test time that value is unknown. We therefore estimate it from observations: from the disagreement among an ensemble of pose estimators, and from frame-to-frame jitter of pose estimates over a short window. Both are cheap and available on real hardware. We also train a version in which the estimator itself is learned from simulated noise, and compare oracle, estimated, and shuffled conditioning.

Baselines

The naive reference point is a diffusion policy trained only on nominal MimicGen-warped data. It should have the best clean precision and the worst noisy success, and it anchors both ends of the frontier.

The strongest existing method is human-seeded Interventional MimicGen. It is the only work in this space that explicitly targets robustness to pose-estimation and geometry errors. The paper reports up to a 39x robustness increase on Nut Insertion, a 24% gain over 100 human interventions on 2-Piece Assembly with only 10 seeds, and 90% on Nut-and-Peg Assembly under geometry error (Interventional MimicGen). We found no reported result on Lacuna for I-MG's clean, noise-free success, so we will measure it ourselves. We will reproduce I-MG with ten seed interventions on the same tasks, the same policy architecture, the same number of synthesized trajectories, and the same training steps as our method. Beating it means matching its noisy success within a few points while recovering at least most of the nominal policy's clean success on insertion tasks.

The mixing-ratio sweep trains unconditioned policies on our autonomous recovery data at varied clean-to-noisy ratios with an equal tuning budget. It is the direct analogue of tuning $\lambda$, and our method must lie strictly above its frontier. A post-hoc clean fine-tuning baseline takes the robust policy and briefly fine-tunes it on nominal data. A two-policy switch runs a nominal and a robust policy and selects between them by the same uncertainty estimate we use for conditioning. This baseline isolates whether shared training matters, or whether the signal alone does the work.

The key ablations keep our method fixed and replace the conditioning input with an oracle value, an estimated value, a constant, and a value shuffled across samples. If the shuffled version performs like the full method, the gain comes from extra data rather than from separating noise levels.

Why not just…?

Why not just tune the mixing ratio of clean and noisy data? This is the most serious alternative because it is free. Our analysis predicts it can only slide along a frontier: any ratio that keeps noisy-input averaging also keeps clean bias. The mixing-ratio sweep in our experiments settles this directly. If some ratio achieves both nominal-level clean precision and I-MG-level noisy success, our conditioning adds nothing and we will report that.

Why not just use architectural equivariance instead of warped data? Equivariant Diffusion Policy builds planar rotational symmetry into the denoising network and reports 21.9% higher success than a standard diffusion policy over 12 MimicGen tasks with 100 demonstrations. However, equivariance handles where the object is, not noise in where the robot thinks it is. A rotated observation with a pose error still yields a rotated wrong action. We include an equivariant backbone as a variant to test whether the clean-precision cost persists with it.

Why not just add standard image augmentation or noise without recovery labels? RoCoDA combines rigid object-pose transforms with crops, hue jitter, and joint noise. It reaches 15.7% versus MimicGen's 9.3% on Three-Piece Assembly, but it never isolates clean precision. Plain input noise is exactly the unconditioned-averaging regime our analysis predicts will bias clean actions, and it provides no recovery behavior. We run a RoCoDA-style noise-augmentation arm to show where it falls on the frontier.

Why not just collect more diverse real data? THE COLOSSEUM reports 30–50% success drops for current policies under single perturbations, including camera pose. This suggests scale and diversity alone do not remove brittleness to perception shifts. Diverse data would also not separate clean from noisy observations, so the same averaging argument applies.

Proposed Experiments

Tasks. We use robosuite tasks from the MimicGen suite, chosen to span tolerance. The tight-tolerance group is Nut Insertion, Nut-and-Peg Assembly, 2-Piece and Three-Piece Assembly. The loose-tolerance group is Three Block Stack and Coffee. Test-time perturbations are Gaussian object-pose noise at four scales plus camera calibration offsets.

Metrics. The metrics are clean success rate, terminal placement error in millimeters on successful and failed clean episodes, success at each noise level, and the area under the success-versus-noise curve. Results are averaged over three seeds with equal compute per arm.

First decisive check. Do unconditioned robust policies lose clean success on tight-tolerance tasks but not on loose ones? Does clean placement error grow with training noise scale in roughly the shrinkage pattern predicted above? If clean success does not drop, the premise fails and the result becomes a negative finding worth reporting: warped recovery data is free.

Second decisive check. Does the estimated-uncertainty conditioned policy dominate the mixing-ratio frontier and the two-policy switch, with the shuffled-conditioning ablation falling back onto that frontier?

Third check. Under camera-pose perturbations from THE COLOSSEUM, does the pose-uncertainty estimate still track actual error, or does the conditioning fail when the noise is structured rather than Gaussian?