Abstract
Multi-objective reinforcement learning (MORL) generalizes standard reinforcement learning (RL) to solve sequential decision-making problems with several, often conflicting, objectives. In such scenarios, no single optimal policy exists; instead, a set of policies must be found to optimize different objective preferences. This paper frames MORL as a meta-learning problem, where the task distribution is defined by preferences over objectives. We demonstrate that this formulation yields a superior approximation of Pareto optimal solutions in terms of both optimality and computational efficiency, particularly in high-dimensional continuous control environments.
The Multi-Objective Challenge
In standard RL, an agent optimizes a single scalar reward. However, real-world applications—such as robotics—frequently involve trade-offs, like balancing movement speed against energy efficiency. The goal of MORL is to identify the Pareto front: the set of optimal policies where improving one objective necessarily degrades another.
Traditional methods often simplify this by using a weighted sum of objectives (linear scalarization). However, finding appropriate weights is difficult, and training separate policies for every weight combination is computationally prohibitive. Most prior research has also been confined to low-dimensional, discrete action spaces. This work addresses these scalability issues for complex, continuous control tasks.
Methodology: MORL as Meta-Learning
The authors propose a framework that treats different reward preferences as distinct "tasks" within a meta-learning setup. By utilizing a first-order approximation of Model-Agnostic Meta-Learning (MAML), the approach learns a "meta-policy" that can rapidly adapt to any specific preference vector.
The training process involves three phases:
- Adaptation Phase: The algorithm samples preference vectors and performs a single update step from the meta-policy to evaluate performance on specific trade-offs.
- Meta-Policy Training Phase: The meta-policy is updated to find an initialization that is "just a few steps away" from being optimal for any given preference.
- Fine-tuning Phase: The trained meta-policy serves as a high-quality starting point, which is then fine-tuned across a range of preferences to construct the final Pareto front.
The underlying updates are performed using Proximal Policy Optimization (PPO).
Experimental Setup
The method was evaluated on five continuous control environments from Roboschool and OpenAI Gym, modified for multi-dimensional rewards:
- Reacher: Balancing distance to target vs. energy cost.
- LunarLanderContinuous: Balancing landing safety vs. fuel consumption.
- Locomotion (HalfCheetah, Ant, Humanoid): High-dimensional tasks (up to 17 DoF) involving speed, energy, joint limits, and collision avoidance.
Key Results and Insights
The meta-learning approach was compared against the Radial Algorithm (RA), a strong baseline that trains policies from random initializations.
- Superior Pareto Fronts: In complex tasks like HalfCheetah, Ant, and Humanoid, the meta-learning approach achieved a higher Hypervolume indicator, signifying a more diverse and optimal set of trade-offs.
- Data Efficiency: The method reached baseline performance levels with significantly less data. For the Humanoid task, it achieved a 10% improvement in the Pareto front while requiring approximately 30,000 fewer iterations during fine-tuning.
- Robustness: The meta-policy developed a more robust "behavioral vocabulary," helping agents avoid local optima (e.g., standing still to avoid falling penalties) that often trap randomly initialized agents.
Discussion and Limitations
While highly effective for complex robots, the overhead of meta-learning may not be justified for simpler tasks like Reacher, where the baseline performed slightly better. Additionally, the current reliance on linear scalarization cannot capture concave regions of the Pareto front. Future work will explore non-linear scalarization and broader sets of robotic tasks to further refine the framework's versatility.
Summary updated