MIMEx: Intrinsic Rewards from Masked Input Modeling

Toru Lin, Allan Jabri

Abstract

Reinforcement learning (RL) in sparse-reward environments remains a significant challenge, often requiring intrinsic motivation to guide exploration. We present MIMEx (Masked Input Modeling for Exploration), a unified framework that treats exploration as a masked reconstruction task. By utilizing a Transformer-based architecture to predict masked portions of trajectory sequences, MIMEx generates a novelty signal that captures temporal dependencies often missed by one-step prediction methods. Our results demonstrate that MIMEx significantly outperforms traditional curiosity-based methods on challenging, sparse-reward robotic manipulation tasks and provides a theoretical bridge between existing exploration strategies.

Introduction

In reinforcement learning, agents typically learn through trial and error, guided by reward signals from the environment. However, in many real-world scenarios—particularly robotic manipulation—rewards are "sparse." An agent may perform thousands of actions before receiving any feedback, making it nearly impossible to learn through random exploration. To address this, researchers employ intrinsic rewards, often referred to as "curiosity," which provide a bonus for visiting novel or surprising states.

Existing curiosity methods, such as Random Network Distillation (RND) or the Intrinsic Curiosity Module (ICM), are often fragmented. Many are "myopic," focusing only on one-step transitions, which limits their ability to capture the complexity of long-term trajectories or high-dimensional pixel data. MIMEx addresses these limitations by framing exploration as a Masked Input Modeling problem, providing a scalable and trajectory-aware approach to defining novelty.

Methodology

MIMEx draws inspiration from successful masked modeling techniques in natural language processing (e.g., BERT) and computer vision (e.g., Masked Autoencoders). The framework operates through four primary stages:

  1. Sequence Encoding: Rather than processing single frames, MIMEx takes a sequence of observations (a trajectory) of length $T$. These observations are encoded into latent embeddings using a Transformer-based architecture.
  2. Online Masking: A portion of these embeddings—empirically determined to be approximately 70%—is randomly masked out during training.
  3. Reconstruction: A Transformer decoder attempts to reconstruct the original, unmasked sequence from the visible fragments.
  4. Intrinsic Reward Generation: The reconstruction loss ($L_t$) serves as the intrinsic reward. A high reconstruction error indicates that the sequence is novel or surprising relative to the agent's prior experience, incentivizing the agent to explore similar states further.

To stabilize the reward signal, MIMEx applies multiple different masks to the same sequence and averages the resulting error, reducing the inherent variance of the stochastic masking process.

A Unified View of Exploration

A key theoretical contribution of MIMEx is the mathematical bridge it builds between masked modeling and pseudo-likelihood. The authors demonstrate that existing methods like RND and ICM are actually restricted cases of this masked prediction framework. By adjusting the mask distribution, MIMEx can emulate these traditional methods or expand beyond them, providing a controllable "knob" for exploration difficulty and focus.

Experimental Evaluation

The performance of MIMEx was evaluated across two primary benchmarks:

  • PixMC-Sparse: A new benchmark introduced by the authors, created by removing dense rewards from the Pixel Motor Control (PixMC) suite. It includes eight robotic manipulation tasks (using Franka and Kuka arms) with egocentric camera views, representing a significant exploration challenge.
  • DeepMind Control Suite (DMC): Six standard continuous control tasks (e.g., cartpole, quadruped) used to verify the method's general applicability.

Key Results

In the PixMC-Sparse benchmark, MIMEx consistently outperformed baselines. In complex "Pick" and "Move" tasks, where traditional methods like ICM and RND often failed to achieve any success (0%), MIMEx reached success rates between 40% and 80%.

Furthermore, the research highlighted the importance of trajectory-level exploration. Using a sequence length greater than 2 significantly improved efficiency, suggesting that encouraging the discovery of novel behaviors is more effective than simply finding novel frames. The method also demonstrated superior sample efficiency and lower variance across random seeds compared to its predecessors.

Discussion and Future Work

While MIMEx excels in sparse-reward settings, it faces common RL challenges, such as a potential performance penalty in very simple environments where exploration is unnecessary. Additionally, the use of a Transformer-based encoder-decoder adds computational overhead compared to simpler distillation methods.

Future research directions include integrating MIMEx with large-scale pre-trained models. Leveraging priors from models trained on extensive video datasets could provide an even more sophisticated understanding of what constitutes an "interesting" or "novel" trajectory in the physical world.

Conclusion

MIMEx provides a robust and unified framework for intrinsic motivation in RL. By shifting the focus from one-step transitions to masked trajectory reconstruction, it offers a more expressive and scalable approach to exploration, particularly in high-dimensional, sparse-reward robotic environments.

Summary updated

BibTeX
@inproceedings{lin2023mimex,
  author = {Toru Lin and Allan Jabri},
  title = {{MIMEx: Intrinsic Rewards from Masked Input Modeling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=g1dMYenhe4},
}
Figure