Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation

Woo Kyung Kim, Minjong Yoo, Honguk Woo

Abstract

Multi-objective Inverse Reinforcement Learning (IRL) typically requires extensive expert demonstrations covering all possible preference trade-offs. ParIRL (Pareto IRL) addresses this "data hunger" by reconstructing the full Pareto frontier from limited expert datasets. The framework employs a recursive reward-distance regularization strategy using the EPIC metric to discover optimal compromise policies and distills them into a preference-conditioned diffusion model for zero-shot adaptation. Experimental results across various control benchmarks, including the CARLA autonomous driving simulator, demonstrate superior frontier coverage and significantly higher training efficiency than existing baselines.

Introduction

In many real-world scenarios, AI agents must balance conflicting objectives, such as speed versus energy efficiency in autonomous vehicles. While Reinforcement Learning (RL) can optimize these trade-offs, it requires precisely defined reward functions. Inverse Reinforcement Learning (IRL) attempts to learn these functions from expert demonstrations. However, existing multi-objective IRL methods struggle when expert data is sparse, often requiring datasets for every possible preference.

Comparison of IRL approaches under limited data constraints.

As illustrated in Figure 1, simply blending limited datasets often results in non-Pareto-optimal policies that fail to achieve high performance on any single objective. ParIRL solves this by identifying the underlying reward structure and "filling in the gaps" to generate a diverse set of expert-level policies from as few as two extreme examples.

Methodology

The ParIRL framework operates in two main phases: recursive discovery and diffusion distillation.

Recursive Reward Distance Regularized IRL

Instead of simple imitation, ParIRL uses a recursive "forking" logic. Starting from two extreme expert datasets (e.g., "Sport" and "Comfort" modes), the algorithm derives new reward functions by maximizing the distance from existing ones in the reward space. This distance is measured using the Equivalent Policy Invariant Comparison (EPIC) metric, which ensures that the discovered rewards lead to distinct, optimal behaviors rather than just numerical variations.

Preference-Conditioned Diffusion Distillation

The discrete policies generated in the first stage are distilled into a single, preference-conditioned diffusion model. This allows the system to interpolate between discovered points on the Pareto frontier, enabling users to specify any arbitrary trade-off at runtime (e.g., "60% speed, 40% efficiency") and generate appropriate actions zero-shot.

The ParIRL pipeline: from extreme expert datasets to diverse custom policies.

Key Results

ParIRL was evaluated on benchmarks including MO-Car, MuJoCo physics tasks, and the CARLA simulator.

  • Superior Frontier Coverage: ParIRL achieved significantly higher Hypervolume (HV) scores—a metric for quality and diversity—compared to baselines like GAIL and AIRL, outperforming them by over 23% in some tasks.
  • Training Efficiency: The recursive approach proved highly efficient, requiring only 13% to 25% of the training timesteps compared to training multiple separate IRL baselines.
  • Zero-Shot Adaptation: In the CARLA simulator, the diffusion-enhanced model successfully generated smooth transitions between driving styles, providing a "Custom Mode" that accurately reflects user-specified preferences without requiring additional expert data.

Conclusion

ParIRL provides a robust solution for multi-objective tasks with limited data. By leveraging reward-space recursion and diffusion-based distillation, it effectively recovers the Pareto frontier, offering a scalable and efficient path toward more adaptable AI agents.

Summary updated

BibTeX
@inproceedings{kim2024pareto,
  author = {Woo Kyung Kim and Minjong Yoo and Honguk Woo},
  title = {{Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure