Abstract
Designing systems where autonomous agents and humans coexist requires optimizing both the number of controllable agents and their underlying reward structures. This paper introduces a hypernetwork-based framework for solving the composition design problem in Partially Controlled Multi-Agent Systems (PCMAS). By leveraging hypernetworks to generate agent policies across a continuous design space, the proposed method avoids the computational burden of repeated training. Evaluated on a large-scale NYC taxi dispatch simulation, the framework achieves near-equilibrium performance and significant system-wide improvements.
Introduction
In many modern infrastructure and service systems, a central designer manages a subset of agents while others operate independently. These Partially Controlled Multi-Agent Systems (PCMAS) are prevalent in autonomous delivery fleets, traffic management, and shared mobility services. A critical challenge for designers is "composition design": determining the optimal number of controllable agents ($N_c$) and their reward parameters ($\alpha_c$) to maximize system utility without destabilizing the environment for independent agents.
The Composition Design Problem
Composition design is inherently a bi-level optimization problem. At the upper level, the designer seeks to maximize a global objective function. At the lower level, agents (both controllable and uncontrollable) engage in a multi-agent reinforcement learning (MARL) process to reach a Nash Equilibrium based on the chosen design parameters. Traditionally, evaluating a single design point requires solving a full MARL problem, which is computationally prohibitive for large-scale systems with high-dimensional design spaces.
Hypernetwork-based Framework
The core innovation of this research is the application of hypernetworks to bypass the need for exhaustive retraining. Instead of training standard policy networks for specific configurations, the authors train a "meta-network"—the hypernetwork.
As shown in Figure 2, the hypernetwork takes the system configuration ($N_c, \alpha_c$) as input and generates the weights for the agents' actor and critic networks. This approach exploits the smoothness of the design space, allowing the model to interpolate policies for unseen configurations. To handle large agent populations, the framework utilizes Mean-Field Reinforcement Learning (MFRL). A key addition is the Mean Action Network, which predicts the current average behavior of the crowd in real-time, significantly improving coordination stability compared to standard MFRL approaches that rely on delayed historical data.
Case Study: NYC Taxi Dispatch
The framework was validated using a real-world dataset from the New York City Taxi & Limousine Commission. The researchers modeled a driver repositioning problem during evening peak hours.
The simulation environment, depicted in Figure 3, consists of a 7x5 grid covering Manhattan and the route to LaGuardia Airport. The system includes "u-agents" (selfish human drivers) and "c-agents" (controllable drivers managed to improve the overall Order Response Rate). The objective was to find the optimal mix of drivers and reward structures that balances service quality with operational profitability.
Results and Insights
The hypernetwork approach demonstrated several key advantages:
- Equilibrium Stability: The method achieved NashConv values near zero, indicating that the generated policies were highly accurate approximations of a Nash Equilibrium. This outperformed traditional multi-layer perceptron (MLP) baselines.
- System Performance: In the NYC simulation, the framework improved the designer’s objective function by up to 13.89% compared to a baseline with no controllable agents.
- Ablation Analysis: The inclusion of the Mean Action Network was crucial; its removal resulted in a 17.46% performance drop, highlighting the importance of real-time crowd prediction in large-scale coordination.
- Optimal Control Utility: The study found that controllable agents provide the most significant benefit when the system is under moderate stress. In extremely low or high demand scenarios, the marginal utility of centralized control decreases.
Conclusion
This research provides a scalable and efficient solution for designing complex multi-agent systems. By using hypernetworks to generalize across design configurations, the framework enables rapid optimization of agent compositions. Future work will focus on expanding the flexibility of reward function optimization and scaling the architecture to handle even more heterogeneous agent types.
Summary updated