Learning Pareto Set for Multi-Objective Continuous Robot Control

Tianye Shu, Ke Shang, Cheng Gong, Yang Nan, Hisao Ishibuchi

Abstract

For a control problem with multiple conflicting objectives, there exists a set of Pareto-optimal policies called the Pareto set instead of a single optimal policy. When a multi-objective control problem is continuous and complex, traditional multi-objective reinforcement learning (MORL) algorithms search for many Pareto-optimal deep policies to approximate the Pareto set, which is quite resource-consuming. In this paper, we propose a simple and resource-efficient MORL algorithm that learns a continuous representation of the Pareto set in a high-dimensional policy parameter space using a single hypernet. The learned hypernet can directly generate various well-trained policy networks for different user preferences. We compare our method with two state-of-the-art MORL algorithms on seven multi-objective continuous robot control problems. Experimental results show that our method achieves the best overall performance with the least training parameters. An interesting observation is that the Pareto set is well approximated by a curved line or surface in a high-dimensional parameter space. This observation will provide insight for researchers to design new MORL algorithms.

Introduction

In robotics, tasks often involve multiple conflicting objectives, such as maximizing speed while minimizing energy consumption. The goal of Multi-Objective Reinforcement Learning (MORL) is to find the Pareto set—a collection of policies where no objective can be improved without degrading another. Traditional MORL approaches typically attempt to find a discrete set of independent policies to approximate this front. However, these methods suffer from two primary bottlenecks: incompleteness (discrete points cannot represent a continuous trade-off curve) and memory bloat (storing numerous deep neural networks is resource-intensive).

Methodology: Hyper-MORL

To address these challenges, we introduce Hyper-MORL, a resource-efficient algorithm that learns a continuous representation of the Pareto set using a single hypernetwork ($H_\phi$). Instead of training robot policies directly, the hypernetwork takes a preference vector $\omega$ as input and generates the weights $\theta$ for a policy network.

The Hyper-MORL framework maps preference vectors to the parameter space, which then determines the performance in the objective space.

A key innovation is the use of a low-dimensional subspace. Rather than generating all $n$ parameters of a policy network freely, the hypernet maps preferences to a small $d$-dimensional subspace ($d \ll n$). Empirical findings suggest that even for complex robotic tasks, a subspace dimension as low as 10 is sufficient to approximate the Pareto set. The training process involves a warm-up stage to find an average policy, followed by a Pareto set learning stage using policy gradients and linear scalarization to cover the range of preferences.

Experimental Evaluation

We evaluated Hyper-MORL on seven multi-objective continuous robot control benchmarks based on the MuJoCo physics engine, including tasks like MO-HalfCheetah and MO-Humanoid. These tasks require balancing forward speed against energy efficiency or jumping height.

Experimental results comparing Hyper-MORL with PG-MORL and PD-MORL across various MuJoCo environments.

Key Results

  • Performance: Hyper-MORL achieved the best average rank (1.57) across all benchmarks based on the Hypervolume (HV) metric, which measures the quality and spread of the Pareto front.
  • Parameter Efficiency: In the 3-objective MO-Hopper-v3 task, Hyper-MORL represented the entire Pareto set with approximately 140,000 parameters, a significant reduction compared to the 29,000,000 parameters required by PG-MORL.
  • Training Speed: Hyper-MORL demonstrates superior training efficiency. For the Humanoid task, it processes in less than one day what preference-conditioned methods like PD-MORL would take an estimated 40 days to achieve.

Discussion and Conclusion

Our findings reveal that the Pareto set in high-dimensional parameter spaces (e.g., >11,000 dimensions) can be effectively approximated by simple curved manifolds in a tiny subspace. This geometric simplicity allows Hyper-MORL to provide continuous coverage of trade-offs, enabling real-time preference adjustment.

While the current implementation uses linear scalarization—which can struggle with concave regions of the Pareto front (as seen in MO-HalfCheetah-v2)—future work will explore integrating non-linear scalarization functions to enhance coverage while maintaining the efficiency of the hypernetwork approach.

Summary updated

Abstract

For a control problem with multiple conflicting objectives, there exists a set of Pareto-optimal policies called the Pareto set instead of a single optimal policy. When a multi-objective control problem is continuous and complex, traditional multi-objective reinforcement learning (MORL) algorithms search for many Pareto-optimal deep policies to approximate the Pareto set, which is quite resource-consuming. In this paper, we propose a simple and resource-efficient MORL algorithm that learns a continuous representation of the Pareto set in a high-dimensional policy parameter space using a single hypernet. The learned hypernet can directly generate various well-trained policy networks for different user preferences. We compare our method with two state-of-the-art MORL algorithms on seven multi-objective continuous robot control problems. Experimental results show that our method achieves the best overall performance with the least training parameters. An interesting observation is that the Pareto set is well approximated by a curved line or surface in a high-dimensional parameter space. This observation will provide insight for researchers to design new MORL algorithms.
BibTeX
@inproceedings{shu2024learning,
  author = {Tianye Shu and Ke Shang and Cheng Gong and Yang Nan and Hisao Ishibuchi},
  title = {{Learning Pareto Set for Multi-Objective Continuous Robot Control}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=jPZoVfmsIx},
}
Figure