Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation

Woo Kyung Kim, Minjong Yoo, Honguk Woo

Introduction

In many real-world autonomous systems, the definition of "optimal" behavior is not singular but depends on user preferences. For instance, an autonomous vehicle might need to balance high speed with energy efficiency or aggressive lane-changing with passenger comfort. Traditional Imitation Learning (IL) often struggles to capture this diversity, typically requiring massive datasets that span the entire spectrum of possible trade-offs. This paper introduces Pareto Inverse Reinforcement Learning (ParIRL), a framework designed to recover the full Pareto frontier of optimal policies from only a few limited expert datasets.

The Challenge of Multi-Objective Expertise

When experts have conflicting objectives, their behaviors represent different points on a Pareto frontier—a set of policies where no objective can be improved without degrading another. Most existing IL methods attempt to handle multiple preferences by "blending" available datasets. However, as illustrated in Figure 1, simply mixing trajectories often leads to sub-optimal, non-Pareto policies that fail to excel at any specific task.

Comparison of dataset types and IRL approaches. ParIRL fills the Pareto frontier where blending fails.

The core research question is: Can we recover the entire spectrum of optimal trade-offs using data from only two experts with extreme preferences?

The ParIRL Framework

ParIRL addresses this through a two-phase process:

  1. Recursive Reward Distance Regularized IRL: The system starts by imitating the two available experts. It then recursively "forks" new training sessions to discover intermediate policies. By using the EPIC metric (Equivalent Policy Invariant Comparison) to constrain the distance between reward functions, the framework ensures that new policies are optimal "neighbors" of existing ones, progressively populating the Pareto frontier. This recursive approach effectively "stretches" the information contained in the initial datasets to cover the entire frontier.
  2. Preference-conditioned Diffusion Model: The discrete policies discovered in the first phase are distilled into a single diffusion model. This model is conditioned on preference weights, allowing for smooth interpolation between learned points and enabling "zero-shot" adaptation to unseen preference ratios. The use of a diffusion model is a key innovation; it captures complex distributions more effectively than standard regression models, ensuring high-quality policy generation across the entire preference space.
Overview of the ParIRL pipeline, from expert modes (Sport/Comfort) to the generation of custom Pareto-optimal policies.

Experimental Evaluation

The authors evaluated ParIRL across several benchmarks, including custom car environments, MuJoCo physics tasks (MO-Ant, MO-Cheetah), and the high-fidelity CARLA simulator.

Key Results

  • Superior Coverage: ParIRL consistently achieved higher Hypervolume (HV) scores than baselines like GAIL or AIRL. For example, in the MO-Ant task, ParIRL reached an HV of 4.99, significantly outperforming AIRL's 4.08. Theoretical analysis also provides regret bounds, ensuring the generated policies stay near-optimal.
  • Density and Smoothness: The diffusion-enhanced model (ParIRL+DU) achieved near-zero sparsity, indicating a dense and continuous frontier of policies rather than a few scattered points.
  • Efficiency: By building on previously learned policies, ParIRL reduced training time to approximately 13%–25% of the time required to train separate baselines for each point on the frontier.
  • Real-World Applicability: In CARLA driving tests, the model successfully generated "Custom" modes that balanced lane-changing frequency and speed, demonstrating its capability to handle complex, high-dimensional sensor data.

Conclusion and Future Work

ParIRL provides a robust method for generating diverse, optimal policies from sparse expert data. By combining recursive reward regularization with diffusion-based distillation, it bridges the gap between limited training data and the need for versatile, preference-aware AI. Future work will explore scaling the framework to handle more than two conflicting objectives and investigating direct learning of continuous preference manifolds.

This summary is based on the research paper "Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation."

Summary updated

Abstract

Data-driven offline reinforcement learning and imitation learning approaches have been gaining popularity in addressing sequential decision-making problems. Yet, these approaches rarely consider learning Pareto-optimal policies from a limited pool of expert datasets. This becomes particularly marked due to practical limitations in obtaining comprehensive datasets for all preferences, where multiple conflicting objectives exist and each expert might hold a unique optimization preference for these objectives. In this paper, we adapt inverse reinforcement learning (IRL) by using reward distance estimates for regularizing the discriminator. This enables progressive generation of a set of policies that accommodate diverse preferences on the multiple objectives, while using only two distinct datasets, each associated with a different expert preference. In doing so, we present a Pareto IRL framework (ParIRL) that establishes a Pareto policy set from these limited datasets. In the framework, the Pareto policy set is then distilled into a single, preference-conditioned diffusion model, thus allowing users to immediately specify which expert's patterns they prefer. Through experiments, we show that ParIRL outperforms other IRL algorithms for various multi-objective control tasks, achieving the dense approximation of the Pareto frontier. We also demonstrate the applicability of ParIRL with autonomous driving in CARLA.
BibTeX
@inproceedings{kim2024pareto,
  author = {Woo Kyung Kim and Minjong Yoo and Honguk Woo},
  title = {{Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure