Pareto Policy Adaptation

Panagiotis Kyriakis, Jyotirmoy Deshmukh, Paul BogdanICLR 2022 · OpenReview

TL;DR

Researchers have developed Pareto Policy Adaptation (PPA), a novel policy gradient method for Multi-Objective Reinforcement Learning (MORL). By leveraging "Pareto stationarity" and implicit differentiation, PPA allows a single reinforcement learning agent to efficiently approximate the entire Pareto front and adapt its behavior to any user preference distribution, avoiding the scalability issues of traditional multi-policy methods.

The Challenge of Multiple Objectives

Most reinforcement learning (RL) algorithms are designed to maximize a single scalar reward. However, real-world problems—like balancing a robot's speed versus its energy consumption—involve multiple conflicting objectives.

The field of Multi-Objective Reinforcement Learning (MORL) typically tackles this in two ways:

  1. Single-policy methods: These collapse multiple rewards into one using fixed weights (preferences). They are simple but brittle; if preferences change, the agent must be retrained from scratch.
  2. Multiple-policy methods: These try to learn an entire set of optimal trade-offs (the Pareto front). While flexible, these methods often struggle to scale to high-dimensional problems because they require maintaining an ensemble of many different policies.

There is a critical gap between these approaches: the field needs a method that is as easy to implement as a single-policy gradient but as adaptable as a multi-policy ensemble.

Introducing Pareto Policy Adaptation

The authors introduce two major components to bridge this gap:

  • Multi-Objective Policy Gradient: Instead of picking arbitrary weights for rewards, the algorithm uses a Projected Gradient Descent (PGD) solver. This solver finds a "common ascent direction"—an update path that improves all objectives simultaneously—by enforcing a first-order condition called Pareto stationarity.
  • Pareto Policy Adaptation (PPA) Loss: This specialized loss function maximizes the scalarized reward while simultaneously penalizing the distance between the "inferred" preferences of the current policy and a "target" preference distribution.
  • Implicit Differentiation: A major technical hurdle in PPA is that the loss depends on the output of an iterative optimization solver (the PGD solver). Differentiating through thousands of solver iterations is computationally expensive. The authors use the Implicit Function Theorem to calculate the gradients "around" the solver, bypassing the need to backpropagate through every internal step. This makes the method compatible with modern deep learning frameworks.

Evaluation Environments

The method was evaluated on four distinct multi-objective environments, modified from standard benchmarks:

  1. Multi-Objective GridWorld (MOGW): A synthetic 16x16 grid where the agent balances reaching a goal vs. a time penalty.
  2. Deep Sea Treasure (DST): A classic MORL benchmark where a submarine searches for treasures of varying values at different depths.
  3. Multi-Objective SuperMario (MOSM): A high-dimensional environment with five rewards: horizontal progress, time, deaths, coins collected, and enemies defeated.
  4. Multi-Objective MuJoCo (MOMU): Continuous control tasks (Ant, Hopper, HalfCheetah, Humanoid) modified to output 2D rewards (e.g., velocity vs. energy consumption).

Key Results

The authors compared PPA against several state-of-the-art baselines using metrics such as HyperVolume (how much of the objective space the Pareto front covers) and Pareto Dominance.

PPA approximating the Pareto front in the Deep Sea Treasure environment.
  • Superior Coverage: In almost every environment, PPA achieved higher HyperVolume scores than baselines. For example, in the MOSM (SuperMario) task, PPA reached a HyperVolume of 1442.14, significantly outperforming the next best baseline (1401.14).
  • Zero-Shot Adaptability: PPA proved it could adapt to different user preferences much more effectively than non-adaptive versions of the same algorithm.
Performance of PPA compared to baselines across MuJoCo continuous control tasks.
  • Dominance and Robustness: The Pareto Dominance metric showed that PPA-trained agents consistently found policies that were non-inferior to those found by other methods. Furthermore, the authors demonstrated that hand-picking fixed weights often leads to poor results or "collapse" in complex environments. PPA’s ability to infer and adapt preferences makes it significantly more robust.

Limitations and Future Work

  • Linear Preferences: The current framework assumes that user preferences are linear combinations of rewards. While common, some real-world preferences might be non-linear.
  • Fine-tuning: While PPA is highly adaptable, if a target preference distribution changes drastically, the model may still require some fine-tuning. Future work could explore ways to leverage the previously learned policy to speed up this transition.
  • Second-Order Derivatives: Implicit differentiation requires computing Jacobians, which can be memory-intensive for extremely large neural networks, though the authors mitigated this using vector-Jacobian products.

Summary updated

Abstract

We present a policy gradient method for Multi-Objective Reinforcement Learning under unknown, linear preferences. By enforcing Pareto stationarity, a first-order condition for Pareto optimality, we...
BibTeX
@inproceedings{kyriakis2022pareto,
  author = {Panagiotis Kyriakis and Jyotirmoy Deshmukh and Paul Bogdan},
  title = {{Pareto Policy Adaptation}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=HgQS-ZKhqO},
}
Figure