Abstract
Multi-objective Inverse Reinforcement Learning (IRL) typically requires extensive expert demonstrations covering all possible preference trade-offs. ParIRL (Pareto IRL) addresses this "data hunger" by reconstructing the full Pareto frontier from limited expert datasets. The framework employs a recursive reward-distance regularization strategy using the EPIC metric to discover optimal compromise policies and distills them into a preference-conditioned diffusion model for zero-shot adaptation. Experimental results across various control benchmarks, including the CARLA autonomous driving simulator, demonstrate superior frontier coverage and significantly higher training efficiency than existing baselines.
Introduction
In many real-world scenarios, AI agents must balance conflicting objectives, such as speed versus energy efficiency in autonomous vehicles. While Reinforcement Learning (RL) can optimize these trade-offs, it requires precisely defined reward functions. Inverse Reinforcement Learning (IRL) attempts to learn these functions from expert demonstrations. However, existing multi-objective IRL methods struggle when expert data is sparse, often requiring datasets for every possible preference.
As illustrated in Figure 1, simply blending limited datasets often results in non-Pareto-optimal policies that fail to achieve high performance on any single objective. ParIRL solves this by identifying the underlying reward structure and "filling in the gaps" to generate a diverse set of expert-level policies from as few as two extreme examples.
Methodology
The ParIRL framework operates in two main phases: recursive discovery and diffusion distillation.
Recursive Reward Distance Regularized IRL
Instead of simple imitation, ParIRL uses a recursive "forking" logic. Starting from two extreme expert datasets (e.g., "Sport" and "Comfort" modes), the algorithm derives new reward functions by maximizing the distance from existing ones in the reward space. This distance is measured using the Equivalent Policy Invariant Comparison (EPIC) metric, which ensures that the discovered rewards lead to distinct, optimal behaviors rather than just numerical variations.
Preference-Conditioned Diffusion Distillation
The discrete policies generated in the first stage are distilled into a single, preference-conditioned diffusion model. This allows the system to interpolate between discovered points on the Pareto frontier, enabling users to specify any arbitrary trade-off at runtime (e.g., "60% speed, 40% efficiency") and generate appropriate actions zero-shot.
Key Results
ParIRL was evaluated on benchmarks including MO-Car, MuJoCo physics tasks, and the CARLA simulator.
- Superior Frontier Coverage: ParIRL achieved significantly higher Hypervolume (HV) scores—a metric for quality and diversity—compared to baselines like GAIL and AIRL, outperforming them by over 23% in some tasks.
- Training Efficiency: The recursive approach proved highly efficient, requiring only 13% to 25% of the training timesteps compared to training multiple separate IRL baselines.
- Zero-Shot Adaptation: In the CARLA simulator, the diffusion-enhanced model successfully generated smooth transitions between driving styles, providing a "Custom Mode" that accurately reflects user-specified preferences without requiring additional expert data.
Conclusion
ParIRL provides a robust solution for multi-objective tasks with limited data. By leveraging reward-space recursion and diffusion-based distillation, it effectively recovers the Pareto frontier, offering a scalable and efficient path toward more adaptable AI agents.
Summary updated