Abstract
Data-driven offline reinforcement learning and imitation learning approaches have been gaining popularity in addressing sequential decision-making problems. Yet, these approaches rarely consider learning Pareto-optimal policies from a limited pool of expert datasets. This becomes particularly marked due to practical limitations in obtaining comprehensive datasets for all preferences, where multiple conflicting objectives exist and each expert might hold a unique optimization preference for these objectives. In this paper, we adapt inverse reinforcement learning (IRL) by using reward distance estimates for regularizing the discriminator. This enables progressive generation of a set of policies that accommodate diverse preferences on the multiple objectives, while using only two distinct datasets, each associated with a different expert preference. In doing so, we present a Pareto IRL framework (ParIRL) that establishes a Pareto policy set from these limited datasets. In the framework, the Pareto policy set is then distilled into a single, preference-conditioned diffusion model, thus allowing users to immediately specify which expert's patterns they prefer. Through experiments, we show that ParIRL outperforms other IRL algorithms for various multi-objective control tasks, achieving the dense approximation of the Pareto frontier. We also demonstrate the applicability of ParIRL with autonomous driving in CARLA.
TL;DR
ParIRL (Pareto Inverse Reinforcement Learning) generates a diverse set of Pareto-optimal policies from just two expert datasets with different preferences. By using a recursive IRL scheme regularized by reward-distance estimates and distilling the results into a preference-conditioned diffusion model, it enables zero-shot adaptation to any user-specified trade-off between conflicting objectives.
The Challenge of Multi-Objective Imitation
In real-world decision-making, agents often face conflicting objectives. For instance, an autonomous vehicle must balance speed against energy efficiency. Different experts exhibit different preferences for these trade-offs. Ideally, training a versatile agent requires a comprehensive dataset covering the entire Pareto frontier—the set of all optimal trade-offs. However, in practice, we often only have access to a few "extreme" datasets (e.g., one very fast driver and one very efficient driver).
Existing imitation learning methods fall short in two ways. First, "Blended IRL" (merging datasets) often results in "average" policies that are sub-optimal for all objectives. Second, most multi-objective algorithms assume the availability of data for the entire frontier, which is rarely the case.
The ParIRL Framework
The authors propose ParIRL, which "fills in the gaps" to generate a dense set of optimal policies from sparse expert examples. The framework operates in two main phases:
1. Recursive Reward Distance Regularized IRL
Instead of direct action imitation, ParIRL uses Inverse Reinforcement Learning (IRL) to infer underlying reward functions. The core novelty is a recursive scheme: starting from two known expert points, the algorithm "crawls" along the Pareto frontier. It generates new policies by regularizing the IRL process using the EPIC metric (Equivalent Policy Invariant Comparison). This regularization ensures that new reward functions are a specific "distance" away from previous ones, allowing the model to systematically discover intermediate trade-offs not present in the original data.
2. Preference-conditioned Diffusion Distillation
The discovered discrete Pareto policies are distilled into a single diffusion model conditioned on a preference vector ($\omega$). Using a technique similar to classifier-free guidance, the model learns to interpolate between the discovered policies. This results in a unified network capable of executing any trade-off on the fly (zero-shot).
Experimental Evaluation
ParIRL was evaluated across several challenging environments:
- MO-MuJoCo: Robotics benchmarks (Swimmer, Cheetah, Ant) balancing speed vs. energy.
- MO-Car: 1D car control with speed and energy objectives.
- CARLA: High-fidelity autonomous driving simulator for "Comfort" vs. "Sport" modes.
Expert datasets were generated using PGMORL, an online RL algorithm, to serve as ground truth for training.
Key Results and Insights
- Superior Frontier Coverage: ParIRL consistently achieved higher Hypervolume (HV)—measuring quality and diversity—than baselines like GAIL, AIRL, and DiffAIL.
- Density and Sparsity: The Sparsity (SP) metric was significantly lower, indicating a much more continuous range of trade-offs.
- The "Blending" Trap: Results confirm that operating in the reward space (via IRL) is essential; simple trajectory mixing leads to sub-optimality.
- Efficiency: The recursive approach reduced training time to 13%–25% of training standard IRL baselines from scratch for each point.
- Zero-Shot Generalization: The distilled diffusion model (ParIRL+DU) outperformed the raw policy set, showing that distillation aids generalization to unseen preference weights.
Conclusion
ParIRL effectively addresses the data scarcity problem in multi-objective imitation learning. By leveraging reward-space regularization and diffusion-based distillation, it enables the generation of diverse, high-quality policies from minimal expert data. Future work may explore scaling to higher-dimensional objectives and handling non-stationary environments.
Summary updated