Learning Pareto Set for Multi-Objective Continuous Robot Control

Tianye Shu, Ke Shang, Cheng Gong, Nan Yang, Hisao Ishibuchi

Abstract

Multi-objective reinforcement learning (MORL) aims to find a set of policies that represent optimal trade-offs between conflicting objectives, such as speed and energy efficiency in robotics. Traditional methods often require training and storing numerous independent policies, leading to high computational and memory costs. We present Hyper-MORL, a novel approach that leverages hypernetworks to learn a continuous low-dimensional manifold of the Pareto set within the high-dimensional policy parameter space. By mapping user preferences to policy weights through a single hypernetwork, Hyper-MORL achieves state-of-the-art performance on complex robotic control tasks with a significant reduction in parameter count compared to discrete policy sets. Our results on MuJoCo-based benchmarks demonstrate that Hyper-MORL provides superior coverage of the Pareto front while maintaining high training efficiency.

What Problem Does This Paper Address?

In real-world robotics, agents rarely have a single goal. Objectives like speed, energy efficiency, and stability often conflict, meaning there isn't one "perfect" policy, but rather a Pareto set of trade-offs. Traditional MORL typically approximates this set by training a finite collection of independent policies. This approach is resource-intensive, requiring the storage of numerous deep neural networks, and often leaves gaps in the trade-off curve where specific user preferences are not represented.

Methods: The Hypernetwork Approach

Hyper-MORL introduces a manifold-based approach using a Hypernetwork ($H_\phi$). Instead of training many separate policies, the framework trains a single "master" network that takes a user's preference vector (e.g., weighting speed vs. energy) as input and outputs the weights for a specific policy network matching that preference.

The Hyper-MORL framework: Mapping preference space to parameter space and finally to objective space.

The core insight is that while a policy might have over 10,000 parameters, the set of optimal policies likely lies on a low-dimensional manifold. Hyper-MORL maps preferences into a small $d$-dimensional subspace (e.g., $d=10$) before projecting them into the full policy parameter space. Training involves a warm-up stage to reach the Pareto front quickly, followed by a Pareto set learning stage using policy gradients (PPO) to map sampled preferences to optimal weights.

Key Results and Efficiency

The researchers evaluated Hyper-MORL on seven multi-objective continuous robot control benchmarks (MuJoCo), including tasks like MO-Ant and MO-Humanoid.

Performance and Coverage

Hyper-MORL achieved the best average rank (1.57) in terms of Hypervolume across all test problems. As shown in the 3-objective MO-Hopper-v3 task, Hyper-MORL provides dense and high-quality coverage of the Pareto front compared to discrete baselines.

Comparison of Hyper-MORL against PG-MORL and PD-MORL on the 3-objective MO-Hopper-v3 task.

Parameter Reduction

The approach offers massive memory savings. In the MO-Ant-v2 task, Hyper-MORL represented the entire Pareto set using roughly 140,000 parameters, whereas the competing PG-MORL required 29,000,000 parameters to store its set of discrete policies. Furthermore, training on complex tasks like Humanoid was completed in less than a day.

Key Insights and Limitations

The success of Hyper-MORL confirms that optimal behavior is highly structured. By sharing knowledge across different trade-offs through the hypernetwork, the system enables smoother transitions between policies. However, the current use of linear scalarization limits the ability to find solutions on concave parts of the Pareto front. Future work will explore non-linear scalarization functions to further improve the expressiveness of the learned manifolds.

Summary updated

BibTeX
@inproceedings{shu2024learning,
  author = {Tianye Shu and Ke Shang and Cheng Gong and Nan Yang and Hisao Ishibuchi},
  title = {{Learning Pareto Set for Multi-Objective Continuous Robot Control}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=jPZoVfmsIx},
}
Figure