Shape-Directed Noise Injection for Imitation Policies That Survive New Object Geometries

Robots trained by imitation, which means copying recorded expert demonstrations, tend to fail in a specific way when handed an object whose shape they have not seen. The first small mistake caused by the new shape, such as a gripper arriving two millimeters off a mug handle, puts the robot in a situation the demonstrations never showed. The next mistake is then larger, and errors pile up until the task fails. A known cure for this pile-up is to add noise to the robot's motions while demonstrations are being recorded, while still recording the expert's corrective action, so that the learner sees how to recover. That noise is usually added equally in every direction, which wastes most of it and can damage hardware. We propose to aim the noise using the object's geometry. In simulation, we randomize object shapes within a category, measure the directions in which a change of shape would move the point the robot must reach, and inject noise mainly along those directions. The result is training data that teaches recovery from exactly the errors that unfamiliar shapes cause. Policies are then trained only on this simulated data and deployed on real robots and real novel objects with no further tuning, which is called zero-shot sim-to-real transfer.

The opportunity comes from an explicit loose end. Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces proves that noise injected during data collection can match interactive expert correction. Its authors note that their isotropic noise is inefficient in high dimensions, and they suggest shaping it along an "excitable manifold" where errors grow fastest. They do not say how to find that manifold. For object manipulation under shape variation, we argue that object geometry largely determines it, and that simulation lets us compute it cheaply.

Core Claim

When an imitation policy fails on a new shape, its closed-loop deviation from the expert is not isotropic. It is concentrated in a low-dimensional subspace given by how the task's interaction points (grasp contacts, hanging points, placement targets) move as the shape varies. If we inject demonstration-time noise mainly in that subspace, we get more recovery coverage per unit of noise than isotropic noise does. That coverage should translate into higher success on held-out and real shapes when the comparison is made at equal noise energy and equal numbers of demonstrations.

Concretely, let $z$ be a low-dimensional shape code for an object category, and let $p_t(z)$ be the 3D interaction point active at time $t$ of a demonstration. The shape Jacobian $J_t = \partial p_t / \partial z$ says how that point moves when the shape changes. We perturb the end-effector command with zero-mean Gaussian noise whose covariance is

$$\Sigma_t = \sigma^2 \left( (1-\lambda)\frac{J_t J_t^\top}{\mathrm{tr}(J_t J_t^\top)} + \frac{\lambda}{3} I \right),$$

where $\sigma^2$ is the total noise energy (held equal across all compared methods), $\lambda \in [0,1]$ mixes in a small isotropic floor, and $I$ is the identity. Setting $\lambda = 1$ recovers the isotropic recipe from the action-chunking paper. The expert's unperturbed corrective action is always the training label.

Research Program

The shape code and the Jacobian come from One-shot Imitation Learning via Interaction Warping. That work builds a category model from about 10 meshes using point-cloud registration plus principal component analysis. It then warps demonstrated interaction points onto new instances by solving for pose and deformation, and it executes the result as an open-loop plan. We reuse its deformation model for two purposes: to sample randomized training shapes, and to differentiate warped interaction points with respect to the shape code. We do not use its slow test-time optimization (25–30 seconds per object) or its open-loop execution. Our deployed policy is a closed-loop point-cloud imitation policy that predicts action chunks and never fits a shape at test time.

The simulated expert is a planner with privileged access to the true mesh and warped interaction points. It generates demonstrations on shapes sampled from the code distribution, and dynamics randomization (mass, friction, controller gains) and depth-sensor noise are applied throughout. Prior work suggests this pattern: Learning Robust Real-World Dexterous Grasping Policies via Implicit Shape Augmentation showed that shape-augmented simulated data can transfer to real objects, reporting 79% success across 22 novel real objects. Its policy, however, predicts a static grasp, so compounding error never arises. Our tasks are multi-step and closed-loop, which is where compounding error lives.

The program has three stages. The first is diagnostic: train plain behavior cloning on randomized shapes, roll it out on shapes outside the training code range, and decompose the deviation from the expert's trajectory into the shape-Jacobian subspace and its complement. If deviation is not concentrated in that subspace, the core claim is false, and we would stop or pivot toward contact-normal shaping. The second stage compares data-collection recipes at matched noise energy. The third stage deploys the best policies zero-shot on a real arm with held-out real objects.

Baselines

The naive reference point is behavior cloning with action chunking, trained on the same randomized-shape simulated demonstrations with no noise injection. For context, What Matters in Learning from Offline Human Demonstrations for Robot Manipulation reports 84% success for its recurrent behavior-cloning model (BC-RNN) on robomimic Square with fixed geometry. We expect our no-noise version to fall well below that once shapes leave the training range, consistent with the 30–50% drops THE COLOSSEUM reports under single perturbations.

The strongest method-level competitor is isotropic noise injection plus action chunking, exactly as in the action-chunking paper and recovered by $\lambda = 1$. It is the strongest competitor because it is the method our proposal modifies, and its authors report it rivals or beats interactive DAgger and DART on MuJoCo tasks. They report gains on robomimic Tool Hang, but no exact success percentage is reported on Lacuna for isotropic noise on Tool Hang, so we will reproduce it ourselves. It receives the same demonstrations, the same total noise energy $\sigma^2$, the same policy architecture, and the same hyperparameter search budget, with $\sigma^2$ swept jointly for both methods. Beating it means higher success on held-out shapes at matched $\sigma^2$, or the same success at substantially lower $\sigma^2$. The second outcome matters for hardware safety.

The strongest geometric competitor is Interaction Warping itself, run open-loop on the same tasks. Its paper reports 82–86% simulated success from one demonstration on Mug on Tree, Bowl on Mug and Bottle in Container, and 86.2% real pick-and-place success. Our closed-loop policy should match it on in-range shapes and exceed it under dynamics perturbation and slip, where it has no recovery mechanism. Its slow test-time fitting is also a cost our policy avoids.

Two ablations isolate the key ingredient. The first replaces the shape Jacobian with a generic geometric direction: noise along the local surface normal and tangent at the interaction point, at equal energy. This tests whether shape variation specifically matters, or just surface geometry. The second uses a random subspace of matched rank, which tests whether any anisotropy helps.

Why not just…?

Why not just use isotropic noise with a larger budget? This is the real threat. If isotropic noise at higher $\sigma^2$ reaches the same held-out success, the contribution reduces to a safety and efficiency gain, not a capability gain. The matched-energy sweep answers this directly, because we report success as a function of $\sigma^2$ for both recipes. The action-chunking paper's own remark that isotropic noise scales poorly with dimension is the reason to expect a gap. The gap should widen on tasks with more degrees of freedom, so we include a bimanual or 6D-orientation task to stress it.

Why not just run DAgger with the simulated expert, since queries are free in simulation? In simulation this is a legitimate upper bound, and we will run it as an oracle. Our recipe's value is that it needs no expert queries from learner-visited states. That property matters when the expert is a human teleoperator or a real-world planner, and the action-chunking paper shows the non-interactive route can match DAgger. We count it a success if shape-directed noise closes most of the gap between no-noise and sim-DAgger at zero additional expert queries. If it closes little of that gap, we will say so.

Why not just randomize shapes more widely or collect more demonstrations? A Smooth Sea Never Made a Skilled SAILOR reports that a Diffusion Policy given 200 demonstrations, about 10 times the data, still fails to match search-based recovery trained on about 20. That suggests coverage of recovery states, not raw volume, is the bottleneck. We test this by giving the no-noise baseline five times our demonstration count and plotting both against held-out shape distance.

Why not just track semantic keypoints, as P3-PO does? P3-PO tracks foundation-model keypoints and reports a 58% improvement over its strongest baseline on novel instances, so it addresses visual shape invariance. It is still behavior cloning on clean demonstrations, so it inherits compounding error. We treat keypoints as a complementary input representation: we will train our recipe with keypoint inputs as well as point clouds, to show that the noise shaping helps regardless of representation.

Proposed Experiments

In simulation, we use the three Interaction Warping tasks, robomimic Square and Tool Hang with category-randomized meshes, and the object-shape and object-size perturbation axes of COLOSSEUM, evaluated with 100 demonstrations per task to match that benchmark's protocol. Each task is evaluated on in-range shapes, on shapes beyond the training code range, and on a held-out set with topological changes such as missing handles or extra prongs. The topological set probes where the Jacobian construction should break down.

The metrics are success rate as a function of distance in shape-code space, success as a function of injected noise energy, and the fraction of closed-loop deviation lying in the Jacobian subspace. That last metric is the diagnostic that must hold before any other result means anything.

On hardware, a single-arm setup runs Mug on Tree and Bottle in Container zero-shot, with 15 or more unseen real objects and 10 trials each. We compare the no-noise, isotropic, shape-directed and Interaction Warping methods, and add a perturbed condition in which the object is nudged mid-task.

The decisive checks are three. First, the deviation decomposition must show concentration in the Jacobian subspace. Second, shape-directed noise must beat isotropic noise at matched energy on beyond-range shapes. Third, the normal/tangent ablation must fall between the two, which would show that information about shape variation, not just surface geometry, drives the gain. If the first check fails, the proposal's premise is wrong. If the second fails while the first holds, the excitable manifold exists but noise shaping is not the right way to exploit it, and a test-time recovery approach in the style of SAILOR becomes the natural next step.