Noise-Guided Transport for Imitation Learning

Lionel Blondé, Joao A. Candido Ramos, Alexandros Kalousis

Abstract

Imitation learning (IL) often struggles in "low-data regimes" where expert demonstrations are scarce. Noise-Guided Transport (NGT) is a lightweight, off-policy IL method designed to overcome these limitations. By framing imitation as an optimal transport problem solved through predicting random noise, NGT achieves expert-level performance on complex continuous control tasks—such as Humanoid locomotion—using as few as 20 expert transitions. Unlike traditional adversarial methods, NGT maintains stability without heavy gradient penalties, utilizing spectral normalization and distributional rewards to provide a robust, sample-efficient learning signal.

Introduction

Most modern imitation learning approaches fall into two categories: Behavioral Cloning (BC), which requires massive datasets to generalize, and Adversarial IL (e.g., GAIL, DAC), which is often unstable and computationally expensive. In many practical applications, such as healthcare or specialized robotics, expert data is a luxury. The challenge lies in creating a method that can learn from a "data diet" without the training instabilities common in GAN-based architectures.

Noise-Guided Transport (NGT) addresses this by reformulating the reward-learning process. Instead of a standard binary classifier, NGT uses an adversarial game centered on noise prediction, providing a solid theoretical foundation in Optimal Transport (OT) theory while remaining computationally efficient.

Methodology

The NGT framework is built upon three core pillars:

  1. Random Priors as Reward: NGT employs a frozen, random "prior" network and a trainable "predictor" network. The agent's goal is to minimize the predictor's error on expert data while maximizing it on its own trajectories. This prediction error serves as a pseudo-density estimator.
  2. Adversarial Optimal Transport: The authors demonstrate that minimizing prediction error on expert data while maximizing it on agent data is mathematically equivalent to minimizing the Earth Mover’s Distance (EMD) in its dual form. This allows the agent to match the expert's distribution effectively.
  3. Stability via Architecture: To avoid the "hacks" like Gradient Penalties (GP) often required for adversarial stability, NGT uses Spectral Normalization and Orthogonal Initialization. For high-dimensional tasks like the Humanoid, it introduces an HL-Gaussian (histogram) loss, treating reward learning as a distributional regression problem to smooth the signal and prevent training collapses.

NGT uses Soft Actor-Critic (SAC) as its underlying RL engine, ensuring the method is fully off-policy and sample-efficient.

Experimental Evaluation

The researchers evaluated NGT across several standard continuous control benchmarks in Gymnasium (MuJoCo) and the DeepMind Control Suite. A primary focus was on data scarcity, testing regimes with as few as 1, 4, or 11 demonstrations, often subsampled to just 20-50 transitions.

Detailed performance of NGT across various MuJoCo environments with 1, 4, and 11 expert demonstrations.

As shown in the detailed task breakdown, NGT (blue line) consistently outperforms or matches state-of-the-art baselines across diverse environments. Notably, it is one of the few methods capable of mastering the Humanoid-v4 task in ultra-low data settings.

Key Results and Insights

NGT demonstrates significant advantages in both performance and efficiency:

  • Sample Efficiency: In the "1 demo" setting, NGT significantly outperformed established baselines like DAC, SAM, and PWIL.
  • Computational Speed: By avoiding expensive gradient penalties and utilizing CUDA Graphs, NGT achieves up to a 3x speedup in training workflows compared to standard adversarial counterparts.
  • Robustness: The use of distributional rewards (HL-Gaussian loss) allows NGT to scale gracefully to high-dimensional tasks where other methods fail due to reward signal instability.
Aggregated performance comparison of NGT against baseline methods across multiple benchmarks.

The aggregated results highlight NGT's superior convergence and final performance compared to the best-performing versions of DAC and SAM, reaching near-expert levels of normalized return much faster and more reliably.

Conclusion

Noise-Guided Transport offers a powerful, lightweight alternative for imitation learning when data is limited. By leveraging the principles of Optimal Transport through a simple noise-prediction objective, it provides a stable and efficient path to expert-level performance in complex environments. Future work may explore extending this "noise-guided" objective to broader generative modeling tasks beyond the scope of reinforcement learning.

Summary updated

BibTeX
@inproceedings{blonde2026noise,
  author = {Lionel Blondé and Joao A. Candido Ramos and Alexandros Kalousis},
  title = {{Noise-Guided Transport for Imitation Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=Smoegnco1C},
}
Figure