Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations

Yilun Hao, Ruinan Wang, Zhangjie Cao, Zihan Wang, Yuchen Cui, Dorsa Sadigh

TL;DR

Robots often struggle to generalize when trained on multimodal data because "extra" sensors (like RGB cameras) can introduce environment-specific noise that doesn't exist in the real world. This paper introduces Masked Imitation Learning (MIL), a bi-level optimization framework that learns a binary mask to automatically filter out distracting modalities, leading to up to a 5.6x performance boost in complex manipulation tasks.

What Problem Does This Paper Address?

In robotics, we often assume that more sensors lead to better performance. However, the authors identify a phenomenon called state over-specification. This occurs when a policy is trained on modalities that are either useless for the task or change significantly between environments (e.g., simulation vs. reality).

For example, a robot trained to pick up a block in simulation might rely on specific RGB pixel values. When moved to a real-world lab with different lighting, that "extra" information becomes a liability, causing the policy to fail. While humans naturally filter out the color of a knife handle when learning to slice an apple, standard imitation learning (IL) models often overfit to these irrelevant details.

Methods

The core of the paper is Masked Imitation Learning (MIL). Instead of just learning a policy, MIL learns which "channels" of information to listen to using a bi-level optimization approach:

  1. The Architecture: The model uses separate feature encoders for each modality (e.g., one for depth, one for joint positions). These features are multiplied by a binary mask ($\Psi$) before being fed into the action predictor.
  2. Inner-Level Optimization: For a fixed mask, the robot learns a policy using standard behavioral cloning (L2 loss) on the training dataset.
  3. Outer-Level Optimization: The algorithm updates the binary mask by evaluating the learned policy on a separate validation dataset. It uses a coordinate descent algorithm to flip bits in the mask, keeping the configuration that minimizes validation loss.
  4. Forward Dynamics for Validation: To combat "compounding errors" (where small mistakes lead the robot far off-track), the authors don't just look at single-step action errors. They learn a forward dynamics model to simulate how the robot would actually move over time, using the distance between these "imagined" trajectories and the ground truth as the validation signal.

Datasets

The authors evaluated MIL across four distinct domains:

  • Reacher: A MuJoCo-based 2D arm task with two settings: one with a targeted distributional shift in goal locations and another adding raw RGB images.
  • Robomimic-Can: A simulated Franka Panda robot picking up cans of different colors.
  • Robomimic-Square: A challenging long-horizon task involving picking up a square nut and placing it on a peg.
  • Bookshelf (Real Robot): A physical Franka Panda robot tasked with reaching a specific section of a bookshelf based on visual cues, where certain camera angles provided "distracting" correlations.

Key Insights and Ideas

  • Binary > Continuous: The authors found that learning a hard binary mask (0 or 1) works significantly better than a "soft" continuous mask. Continuous masks often fail to completely zero out harmful modalities, allowing noise to leak through.
  • Trajectory-Based Validation is Crucial: In long-horizon tasks like Robomimic-Square, simple per-step action loss is a poor predictor of success. Using a forward dynamics model to evaluate full trajectories in the outer loop was essential for finding the right mask.
  • Recovering Invariance: MIL successfully "discovered" which modalities were environment-invariant. In the real-robot task, it correctly identified that the side-view camera was reliable while the top-down view contained misleading correlations, matching the "Oracle" mask designed by humans.

Key Results

  • Massive Performance Gains: In the Robomimic-Square task, MIL achieved a 71.3% success rate, a 5.6x improvement over vanilla Behavioral Cloning (12.7%).
  • Sim-to-Real Robustness: On the real-world Bookshelf task, MIL achieved a 95.24% success rate, whereas standard BC (using all modalities) lagged at 54.17%.
  • Efficiency: MIL outperformed other common regularization techniques like MaskDropout (randomly masking during training) and ContinuousMask (using SGD on soft weights).

Limitations and Future Work

  • Computational Cost: Because it uses bi-level optimization, MIL is computationally expensive. The inner loop (training a full policy) must be run multiple times to update the mask.
  • Modality Ordering: The current coordinate descent approach updates the mask bit-by-bit. The authors noted that the order in which modalities are evaluated can affect the final mask, potentially missing the global optimum if modalities have complex inter-dependencies.
  • Static Masks: Currently, the mask is fixed for the entire task. Future work could explore dynamic masking, where the robot selectively attends to different sensors at different stages of a multi-step task.

Summary updated

Abstract

Multimodal demonstrations provide robots with an abundance of information to make sense of the world. However, such abundance may not always lead to good performance when it comes to learning sensorimotor control policies from human demonstrations. Extraneous data modalities can lead to state over-specification, where the state contains modalities that are not only useless for decision-making but also can change data distribution across environments. State over-specification leads to issues such as the learned policy not generalizing outside of the training data distribution. In this work, we propose Masked Imitation Learning (MIL) to address state over-specification by selectively using informative modalities. Specifically, we design a masked policy network with a binary mask to block certain modalities. We develop a bi-level optimization algorithm that learns this mask to accurately filter over-specified modalities. We demonstrate empirically that MIL outperforms baseline algorithms in simulated domains and effectively recovers the environment-invariant modalities on a multimodal dataset collected on a real robot. Videos and supplemental details are at: https://tinyurl.com/masked-il
BibTeX
@inproceedings{hao2023masked,
  author = {Yilun Hao and Ruinan Wang and Zhangjie Cao and Zihan Wang and Yuchen Cui and Dorsa Sadigh},
  title = {{Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=y7jybpdymy},
}
Figure