Meta-Learning for Multi-objective Reinforcement Learning

Xi Chen, Ali Ghadirzadeh, Mårten Björkman, Patric Jensfelt

Abstract

Multi-objective reinforcement learning (MORL) generalizes standard reinforcement learning (RL) to solve sequential decision-making problems with several, often conflicting, objectives. In such scenarios, no single optimal policy exists; instead, a set of policies must be found to optimize different objective preferences. This paper frames MORL as a meta-learning problem, where the task distribution is defined by preferences over objectives. We demonstrate that this formulation yields a superior approximation of Pareto optimal solutions in terms of both optimality and computational efficiency, particularly in high-dimensional continuous control environments.

The Multi-Objective Challenge

In standard RL, an agent optimizes a single scalar reward. However, real-world applications—such as robotics—frequently involve trade-offs, like balancing movement speed against energy efficiency. The goal of MORL is to identify the Pareto front: the set of optimal policies where improving one objective necessarily degrades another.

Conceptual overview of MORL and the Pareto front across different robotic tasks.

Traditional methods often simplify this by using a weighted sum of objectives (linear scalarization). However, finding appropriate weights is difficult, and training separate policies for every weight combination is computationally prohibitive. Most prior research has also been confined to low-dimensional, discrete action spaces. This work addresses these scalability issues for complex, continuous control tasks.

Methodology: MORL as Meta-Learning

The authors propose a framework that treats different reward preferences as distinct "tasks" within a meta-learning setup. By utilizing a first-order approximation of Model-Agnostic Meta-Learning (MAML), the approach learns a "meta-policy" that can rapidly adapt to any specific preference vector.

The training process involves three phases:

  1. Adaptation Phase: The algorithm samples preference vectors and performs a single update step from the meta-policy to evaluate performance on specific trade-offs.
  2. Meta-Policy Training Phase: The meta-policy is updated to find an initialization that is "just a few steps away" from being optimal for any given preference.
  3. Fine-tuning Phase: The trained meta-policy serves as a high-quality starting point, which is then fine-tuned across a range of preferences to construct the final Pareto front.

The underlying updates are performed using Proximal Policy Optimization (PPO).

Experimental Setup

The method was evaluated on five continuous control environments from Roboschool and OpenAI Gym, modified for multi-dimensional rewards:

  • Reacher: Balancing distance to target vs. energy cost.
  • LunarLanderContinuous: Balancing landing safety vs. fuel consumption.
  • Locomotion (HalfCheetah, Ant, Humanoid): High-dimensional tasks (up to 17 DoF) involving speed, energy, joint limits, and collision avoidance.

Key Results and Insights

The meta-learning approach was compared against the Radial Algorithm (RA), a strong baseline that trains policies from random initializations.

Comparison of performance (Hypervolume indicator) between the proposed meta-learning approach (blue) and the baseline (red) across environments.
  • Superior Pareto Fronts: In complex tasks like HalfCheetah, Ant, and Humanoid, the meta-learning approach achieved a higher Hypervolume indicator, signifying a more diverse and optimal set of trade-offs.
  • Data Efficiency: The method reached baseline performance levels with significantly less data. For the Humanoid task, it achieved a 10% improvement in the Pareto front while requiring approximately 30,000 fewer iterations during fine-tuning.
  • Robustness: The meta-policy developed a more robust "behavioral vocabulary," helping agents avoid local optima (e.g., standing still to avoid falling penalties) that often trap randomly initialized agents.

Discussion and Limitations

While highly effective for complex robots, the overhead of meta-learning may not be justified for simpler tasks like Reacher, where the baseline performed slightly better. Additionally, the current reliance on linear scalarization cannot capture concave regions of the Pareto front. Future work will explore non-linear scalarization and broader sets of robotic tasks to further refine the framework's versatility.

Summary updated

Abstract

Multi-objective reinforcement learning (MORL) is the generalization of standard reinforcement learning (RL) approaches to solve sequential decision making problems that consist of several, possibly conflicting, objectives. Generally, in such formulations, there is no single optimal policy which optimizes all the objectives simultaneously, and instead, a number of policies has to be found each optimizing a preference of the objectives. In other words, the MORL is framed as a meta-learning problem, with the task distribution given by a distribution over the preferences. We demonstrate that such a formulation results in a better approximation of the Pareto optimal solutions in terms of both the optimality and the computational efficiency. We evaluated our method on obtaining Pareto optimal policies using a number of continuous control problems with high degrees of freedom.
BibTeX
@inproceedings{chen2019meta,
  author = {Xi Chen and Ali Ghadirzadeh and Mårten Björkman and Patric Jensfelt},
  title = {{Meta-Learning for Multi-objective Reinforcement Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=8NQ2hJ1kvcL},
}
Figure