Reward-free World Models for Online Imitation Learning

Shangzhe Li, Zhiao Huang, Hao Su

TL;DR

IQ-MPC is a novel framework for online imitation learning that leverages "reward-free" world models to master complex, high-dimensional robotics tasks. By learning environment dynamics entirely in latent space and deriving rewards directly from a learned Q-function, the method avoids the instability of traditional adversarial learning and achieves expert-level performance on challenging benchmarks like dexterous hand manipulation and high-dimensional locomotion.

What Problem Does This Paper Address?

Imitation Learning (IL) allows robots to learn skills from expert demonstrations rather than trial-and-error reward signals. While simple "Behavioral Cloning" (BC) works for basic tasks, it suffers from bias accumulation: small errors lead the robot into unfamiliar states where it doesn't know what to do, causing total failure.

Online IL addresses this by letting the agent interact with the environment to explore. However, current online methods (like GAIL or IQ-Learn) often struggle with:

  1. High-Dimensionality: Processing raw pixels or controlling complex robots (like a 38-joint dog) is computationally taxing.
  2. Training Instability: Many methods use a "min-max" optimization (similar to GANs) between a reward-giver and a policy-follower, which is notoriously difficult to balance.
  3. Complex Dynamics: Model-free agents often lack a "deep" understanding of how their actions affect the future, making planning difficult in intricate environments.

Methods

The authors introduce IQ-MPC (Inverse Soft-Q Learning for Model Predictive Control). The architecture consists of four main components: an encoder (mapping observations to latent space), latent dynamics (predicting future latent states), a Q-function (critic), and a policy prior.

Key innovations include:

  • Reward-free World Model: Unlike standard world models that predict external rewards, IQ-MPC learns dynamics entirely in a latent space without a decoder (no pixel reconstruction). It eliminates the need for a separate reward model by "decoding" rewards directly from the Q-function using the inverse Bellman operator.
  • Q-Policy Space Optimization: Instead of the unstable reward-policy optimization used in Inverse Reinforcement Learning, the authors reformulate the objective in the Q-policy space. This leverages the Inverse Soft-Q Learning objective to make training significantly more stable.
  • Latent Planning with MPPI: For control, the agent doesn't just react; it plans. It uses Model Predictive Path Integral (MPPI) to simulate thousands of possible future trajectories in the latent space and chooses the best sequence of actions based on the rewards derived from its own Q-function.
Figure 1. IQ-MPC We demonstrate the training workflow for IQ-MPC. The reward-free world model leverages both expert and behavioral data for training, using objectives in Section 4.1. The policy prior from the world model guides the MPPI planning process along with rewards decoded from Q estimations. The detailed planning process is revealed in Algorithm 1.

Datasets

The researchers evaluated IQ-MPC on three diverse and challenging suites:

  • DeepMind Control Suite (DMControl): Locomotion tasks ranging from simple (Hopper) to highly complex (Humanoid, Dog).
  • MyoSuite: Physiologically accurate musculoskeletal models requiring dexterous hand control (e.g., Twirling a pen, Turning a key).
  • ManiSkill2: General robotic manipulation tasks like picking and lifting cubes.

Key Insights and Ideas

  • The Q-Function is the Reward: The most elegant "aha moment" is realizing that if you have a well-learned Q-function and a policy, the reward signal is mathematically implicit. You don't need a separate discriminator to tell the agent it's doing a good job; the Q-function already contains that information.
  • Efficiency via Latent Space: By avoiding the need to reconstruct images (decoder-free), the world model can focus entirely on the features relevant for control. This makes it much more efficient for high-dimensional visual tasks.
  • Stability through Regularization: To prevent the "critic" (Q-function) from becoming too powerful and overwhelming the policy, the authors successfully applied a Wasserstein-1 gradient penalty, a technique borrowed from stable GAN training.

Key Results

IQ-MPC consistently outperformed state-of-the-art baselines (like IQL+SAC and HyPE) in both performance and stability.

  • High-Dimensional Success: In the "Dog Run" and "Humanoid Walk" tasks—environments where most imitation learning agents fail to even stand up—IQ-MPC achieved near-expert performance.
  • Dexterity: In MyoSuite's "Pen Twirl" task, IQ-MPC reached a success rate of 73%, while all other baselines remained at 0%.
  • Sample Efficiency: The model can reach expert-level performance with as few as 10 expert demonstrations in some tasks, as shown in their ablation studies.
Locomotion results showing IQ-MPC (red) reaching expert levels (dotted green) while others struggle

Limitations and Future Work

While powerful, the authors note a few hurdles:

  • Sensitivity to Scale: In very low-dimensional environments, the "discriminator" effect of the Q-function can become too strong, leading to instability.
  • Computational Cost: Planning with MPC is more computationally expensive than using a simple reactive neural network policy during inference.
  • Stochasticity: While the model showed some robustness to noise, it was primarily designed for deterministic environments. Extending this to highly stochastic, real-world settings is a key next step.

The authors suggest that applying this method to real-world physical robots is the ultimate frontier to test its generalization and practical effectiveness.

Summary updated

Abstract

Imitation learning (IL) enables agents to acquire skills directly from expert demonstrations, providing a compelling alternative to reinforcement learning. However, prior online IL approaches struggle with complex tasks characterized by high-dimensional inputs and complex dynamics. In this work, we propose a novel approach to online imitation learning that leverages reward-free world models. Our method learns environmental dynamics entirely in latent spaces without reconstruction, enabling efficient and accurate modeling. We adopt the inverse soft-Q learning objective, reformulating the optimization process in the Q-policy space to mitigate the instability associated with traditional optimization in the reward-policy space. By employing a learned latent dynamics model and planning for control, our approach consistently achieves stable, expert-level performance in tasks with high-dimensional observation or action spaces and intricate dynamics. We evaluate our method on a diverse set of benchmarks, including DMControl, MyoSuite, and ManiSkill2, demonstrating superior empirical performance compared to existing approaches.
BibTeX
@inproceedings{li2025reward,
  author = {Shangzhe Li and Zhiao Huang and Hao Su},
  title = {{Reward-free World Models for Online Imitation Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=owEhpoKBKC},
}
Figure