Learning Pareto Set for Multi-Objective Continuous Robot Control

Tianye Shu, Ke Shang, Cheng Gong, Yang Nan, Hisao Ishibuchi

Abstract

For a control problem with multiple conflicting objectives, there exists a set of Pareto-optimal policies called the Pareto set instead of a single optimal policy. When a multi-objective control problem is continuous and complex, traditional multi-objective reinforcement learning (MORL) algorithms search for many Pareto-optimal deep policies to approximate the Pareto set, which is quite resource-consuming. In this paper, we propose a simple and resource-efficient MORL algorithm that learns a continuous representation of the Pareto set in a high-dimensional policy parameter space using a single hypernet. The learned hypernet can directly generate various well-trained policy networks for different user preferences. We compare our method with two state-of-the-art MORL algorithms on seven multi-objective continuous robot control problems. Experimental results show that our method achieves the best overall performance with the least training parameters. An interesting observation is that the Pareto set is well approximated by a curved line or surface in a high-dimensional parameter space. This observation will provide insight for researchers to design new MORL algorithms.

Introduction

In robot control, agents rarely pursue a single goal. A walking robot, for instance, must balance forward speed with energy efficiency and stability. These objectives are often in conflict: increasing speed typically degrades energy efficiency. In Multi-Objective Reinforcement Learning (MORL), the objective is to identify the Pareto set—the collection of policies where no single objective can be improved without compromising another.

Traditional MORL approaches face two significant hurdles. First is memory bloat: they often require training and storing hundreds of individual deep neural networks to approximate the Pareto set. Second is discontinuity: they provide a finite set of discrete solutions, making it impossible to retrieve a policy for a specific trade-off that wasn't explicitly trained. Hyper-MORL addresses these by learning a continuous manifold of optimal policies.

Methods: The Hyper-MORL Framework

Hyper-MORL shifts the paradigm from learning individual policies to learning a continuous representation of the Pareto set in the high-dimensional policy parameter space using a single hypernetwork.

Hypernetwork Architecture

Instead of a standard policy network, Hyper-MORL employs a hypernetwork $\mathcal{H}$ that takes a user's preference vector $\omega$ (e.g., a weighting between speed and energy) as input and outputs the weights $\theta$ for a target policy network. The framework assumes the Pareto set resides on a low-dimensional manifold. The hypernet maps preferences to a small $d$-dimensional subspace ($d=10$) before projecting them into the massive $n$-dimensional space of robot control parameters.

Overview of the Hyper-MORL framework mapping preference space to parameter space and then to objective space.

Two-Stage Training

  1. Warm-up Stage: The hypernet is initialized to a "middle-of-the-road" policy that performs reasonably well across all objectives, providing a stable starting point.
  2. Pareto Set Learning Stage: The algorithm uses a gradient-based approach. In each iteration, it samples random preferences, generates corresponding policies via the hypernet, and uses Proximal Policy Optimization (PPO) to update the hypernet's weights based on the scalarized reward $\omega^T J(\theta)$.

Experimental Setup

The method was evaluated on seven multi-objective continuous robot control benchmarks using the MuJoCo physics engine. These include 2-objective tasks (Swimmer, HalfCheetah, Walker2d, Ant, Hopper, and Humanoid) and a 3-objective task (Hopper-v3). These benchmarks test the ability to manage trade-offs between velocity, energy consumption, and posture.

Key Results

Hyper-MORL achieved the best average rank in terms of Hypervolume (HV)—a metric measuring the quality and diversity of the Pareto set—across the seven benchmarks.

  • Massive Parameter Efficiency: In the MO-Hopper-v3 task, Hyper-MORL represented the entire Pareto set using approximately 140,000 parameters. In contrast, the state-of-the-art PG-MORL required 29,000,000 parameters to store its archive of discrete policies—a 200x reduction in memory requirements.
  • Continuous Frontier: Unlike evolutionary methods that produce discrete points, Hyper-MORL generates a continuous frontier, allowing for the retrieval of a well-trained policy for any arbitrary preference vector.
Comparison of Pareto fronts for the MO-Walker2d-v2 task, showing Hyper-MORL's competitive performance with significantly fewer parameters.

Discussion and Conclusion

A striking insight from this work is that the Pareto set, despite existing in a massive parameter space, often resides on a simple, low-dimensional curved line or surface. This simplicity allows a single hypernet to capture the entire range of optimal behaviors.

Currently, Hyper-MORL uses linear scalarization, which can struggle with "concave" parts of the Pareto front (as seen in the MO-HalfCheetah-v2 results). Future work will explore non-linear scalarization functions, such as the Tchebycheff approach, to better handle complex, non-convex Pareto fronts and scale the method to even higher-dimensional real-world robotic systems.

Summary updated

Abstract

For a control problem with multiple conflicting objectives, there exists a set of Pareto-optimal policies called the Pareto set instead of a single optimal policy. When a multi-objective control problem is continuous and complex, traditional multi-objective reinforcement learning (MORL) algorithms search for many Pareto-optimal deep policies to approximate the Pareto set, which is quite resource-consuming. In this paper, we propose a simple and resource-efficient MORL algorithm that learns a continuous representation of the Pareto set in a high-dimensional policy parameter space using a single hypernet. The learned hypernet can directly generate various well-trained policy networks for different user preferences. We compare our method with two state-of-the-art MORL algorithms on seven multi-objective continuous robot control problems. Experimental results show that our method achieves the best overall performance with the least training parameters. An interesting observation is that the Pareto set is well approximated by a curved line or surface in a high-dimensional parameter space. This observation will provide insight for researchers to design new MORL algorithms.
BibTeX
@inproceedings{shu2024learning,
  author = {Tianye Shu and Ke Shang and Cheng Gong and Yang Nan and Hisao Ishibuchi},
  title = {{Learning Pareto Set for Multi-Objective Continuous Robot Control}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=jPZoVfmsIx},
}
Figure