Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations

Yilun Hao, Ruinan Wang, Zhangjie Cao, Zihan Wang, Yuchen Cui, Dorsa Sadigh

Introduction: The Curse of Too Much Data

In robotics, the prevailing wisdom often suggests that more sensors lead to more robust policies. However, this paper identifies a critical failure mode called state over-specification. When a policy is trained on multimodal data (e.g., RGB, depth, and proprioception), it may inadvertently rely on environment-specific noise—such as specific lighting or background textures in a simulation—that does not generalize to the real world.

Masked Imitation Learning (MIL) filters out distracting modalities to improve generalization from simulation to real-world environments.

While humans can easily distinguish between task-relevant information (the shape of an object) and irrelevant details (the color of a table), standard imitation learning (IL) models often overfit to these distractions. This leads to a significant performance drop when the robot is deployed in a new environment.

Masked Imitation Learning (MIL)

To address this, the authors propose Masked Imitation Learning (MIL), a framework designed to automatically discover which modalities are truly invariant across environments. MIL employs a bi-level optimization strategy to learn a binary mask ($\Psi$) that selectively filters input features before they reach the policy network.

The MIL framework uses bi-level optimization: an inner loop trains the policy, while an outer loop updates the modality mask based on validation trajectory loss.

Bi-Level Optimization

The MIL process consists of two nested loops:

  1. Inner-Level Optimization: For a given mask, the model learns a policy ($F_\theta$) by minimizing a standard behavioral cloning loss ($\mathcal{L}_{in}$) on the training dataset.
  2. Outer-Level Optimization: The binary mask is updated by evaluating the learned policy on a separate validation dataset. Instead of using simple per-step action errors, MIL utilizes a forward dynamics model ($\bar{T}$) to simulate full trajectories. The validation loss ($\mathcal{L}_{out}$) is calculated based on the distance between these "imagined" rollouts and the ground truth trajectories. This approach better accounts for compounding errors in long-horizon tasks.

The algorithm uses coordinate descent to iteratively flip bits in the binary mask, retaining configurations that improve validation performance. The authors found that hard binary masks significantly outperform soft, continuous masks, which often allow harmful noise to leak through.

Experimental Results and Insights

The authors evaluated MIL across several domains, including simulated manipulation tasks (Robomimic-Can and Square) and a real-world bookshelf reaching task.

  • Significant Performance Boosts: In the complex Robomimic-Square task, MIL achieved a 71.3% success rate, a 5.6x improvement over vanilla Behavioral Cloning (12.7%).
  • Sim-to-Real Robustness: On a physical Franka Panda robot, MIL reached a 95.24% success rate by correctly identifying and masking out unreliable camera views, whereas standard BC lagged at 54.17%.
  • Discovery of Invariance: MIL successfully "discovered" oracle-like masks. For instance, it learned to ignore RGB data when depth and proprioception were sufficient and more stable across environments.

Conclusion and Future Directions

Masked Imitation Learning provides a principled way to handle multimodal data in robotics by filtering out distracting information. While the current bi-level optimization approach is computationally intensive and uses a static mask, it represents a major step toward building robots that can truly generalize. Future research may explore dynamic masking—where the robot adjusts its attention to different sensors at different stages of a task—and more efficient optimization techniques.

Summary updated

BibTeX
@inproceedings{hao2023masked,
  author = {Yilun Hao and Ruinan Wang and Zhangjie Cao and Zihan Wang and Yuchen Cui and Dorsa Sadigh},
  title = {{Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=y7jybpdymy},
}
Figure