TL;DR
InneR (Inner Product Regularization) is a new technique that makes imitation learning policies significantly more robust by enforcing consistent responses to state perturbations. Instead of simply dampening a robot's sensitivity to all inputs, InneR aligns the "null space" of the policy across local neighborhoods, allowing the robot to ignore irrelevant distractors while remaining highly responsive to task-critical information.
What Problem Does This Paper Address?
Imitation learning—where a robot learns a task from human demonstrations—often suffers from covariate shift. Small execution errors compound, pushing the robot into states it never saw during training. In these unfamiliar states, the robot might latch onto irrelevant features (like the position of a background object) rather than the actual goal, a phenomenon known as causal confusion.
Standard regularization techniques, like penalizing the Frobenius or nuclear norm of the policy Jacobian, try to fix this by making the policy less sensitive to every change in the state. However, this is a blunt instrument: it dampens the robot's response to important task-relevant changes just as much as it dampens noise. The authors argue that a robust policy shouldn't just be "less sensitive"; it should be consistently sensitive to the same directions in state space across local regions.
Methods
The core of InneR is a geometric approach to policy sensitivity.
- The Pullback Metric: The authors show that an inner product on the action space (measuring how different two actions are) induces an inner product on the state space through the policy Jacobian. For stochastic policies (like Diffusion Policies), this induced metric is the Fisher Information Matrix (FIM).
- Null Space Alignment: The "null space" of this matrix represents directions in the state space that the policy ignores. If a perturbation (like moving a distractor object) falls in this null space, the robot's action doesn't change.
- Clustering and Averaging: InneR doesn't enforce a global null space (which would be too restrictive). Instead, it clusters states in a training batch that have similar sensitivity directions.
- Eigenvalue Minimization: For each cluster, InneR calculates the average inner product matrix and minimizes its smallest eigenvalues. This forces the states within that cluster to share a common null space—essentially training the robot to ignore the same irrelevant dimensions across a local neighborhood.
Datasets
The authors evaluated InneR across three distinct environments:
- Single-Integrator Simulation: A point-mass goal-reaching task used to test robustness against increasing dimensions of irrelevant noise and train-test distribution shifts.
- Drone Landing Simulation: A Crazyflie quadrotor landing task in a physics simulator (
gym-pybullet-drones), where distractor drones were added to the state to test robustness in nonlinear dynamics. - Hardware Experiments: A 7-DoF Franka Emika Panda robot performing three tasks: "closing a coffee maker," "stacking two cubes," and "pushing a pot." These tasks included real-world distractors like fruit (apples, oranges, bananas) that were irrelevant to the goal.
Key Insights and Ideas
- Direction > Magnitude: The "aha moment" is that what a robot ignores is more important than how much it ignores. By regularizing the null space rather than the norm, InneR preserves the policy's expressiveness in important directions.
- Local Consistency: A robot's focus changes during a task. When picking up a block, the block's pose is critical; once it's in the gripper, that pose becomes constant relative to the hand, and the target location becomes the new focus. InneR’s clustering approach allows the set of "ignored directions" to shift fluidly as the task progresses.
- Fisher Information as a Bridge: The paper provides a elegant theoretical link showing that Fisher Information is the stochastic equivalent of the Jacobian-based state-space metric used for deterministic policies.
Key Results
- Real-World Success: On the Franka Panda robot, InneR raised the average success rate of a Diffusion Policy from 18% to 57% in scenarios with multiple distractors, without requiring any additional training data.
- Robustness to Noise: In drone landing tasks, InneR maintained high success rates even as the state dimension grew with irrelevant distractor data, whereas standard Behavior Cloning (BC) performance dropped significantly.
- Selectivity: Quantitative analysis showed InneR was 6.5 to 8.4 times more sensitive to task-relevant objects than to distractors, significantly outperforming both BC and standard norm-based regularizers.
Limitations and Future Work
- Computational Overhead: InneR is computationally expensive, increasing training time by roughly 5x to 6x. This is due to the need for second-order derivatives (differentiating through the Jacobian) and performing eigenvalue decompositions on every batch.
- High-Dimensional Inputs: The current implementation was tested on state-vector inputs (e.g., object poses). The authors note that extending this to high-dimensional raw inputs like images or point clouds is a major challenge, as computing the full Jacobian for such inputs is currently intractable. Future work may involve using efficient Jacobian-norm estimators to scale InneR to vision-based policies.
Summary updated