M^3RL: Mind-aware Multi-agent Management Reinforcement Learning

Tianmin Shu, Yuandong Tian

Abstract

M^3RL (Mind-aware Multi-agent Management Reinforcement Learning) introduces a hierarchical framework where a manager agent coordinates self-interested worker agents by inferring their hidden preferences and skills. By offering personalized contracts—combinations of specific goals and bonuses—the manager learns to align individual incentives with global productivity, enabling effective ad-hoc teaming even when worker capabilities change over time.

Introduction

In multi-agent reinforcement learning (MARL), the standard assumption is often that agents are either fully cooperative or centrally controlled. However, real-world systems—from economic markets to warehouse robotics—frequently involve self-interested "worker" agents with private preferences and heterogeneous skills. This creates a "principal-agent" problem characterized by information asymmetry (not knowing worker capabilities) and incentive misalignment (workers preferring easier tasks over high-priority ones).

The M^3RL framework addresses this by introducing a hierarchical structure where a "manager" agent coordinates these workers. Instead of direct control, the manager influences behavior by offering personalized contracts: specific goals paired with monetary bonuses.

The M^3RL problem setup: (a) Workers have diverse skills and preferences; (b) The manager faces incomplete information; (c) The manager generates contracts to align incentives.

Methodology

The core of M^3RL is the manager's ability to infer the hidden "minds" of workers to generate effective contracts. The architecture is composed of three specialized modules:

  1. Performance History Module: This module maintains a "resume" for each worker by encoding their past success rates across various tasks and bonus levels. This allows the manager to identify and remember worker capabilities across different episodes.
  2. Mind Tracker Module: Acting as the framework's "empathy engine," this module uses an LSTM to track worker behavior within an episode. It is trained via Imitation Learning (IL) to predict a worker's next action, forcing the internal representation to capture the worker's current intentions and skills.
  3. Manager Module: This module outputs the final goal and bonus policy. It utilizes Successor Representations (SR) to disentangle the expected future rewards (goal achievement) from the expected costs (bonus payments). This separation is critical for learning in environments with sparse rewards or complex task dependencies.
The M^3RL network architecture, illustrating the integration of performance history, mind tracking, and the manager's policy generation using Successor Representations.

Experimental Evaluation

The framework was tested in two 2D grid-world environments designed to mimic complex management problems:

  • Resource Collection: Workers collect resources based on varying digging skills and preferences. The manager must learn which worker is most efficient for specific resource types.
  • Crafting (Hierarchical Task): A complex environment where workers must craft intermediate items to eventually produce high-value "top-level" items. This tests the manager's ability to plan sequences and invest bonuses in preparatory tasks.

Key Insights

  • Incentives as Coordination: M^3RL demonstrates that coordination can be achieved through contract design rather than direct control, allowing for effective management of independent agents.
  • Theory of Mind via Imitation: Using an auxiliary imitation loss is a powerful way to learn latent worker characteristics. By learning to "mimic" the worker, the manager naturally develops a model of their skill and preference profile.
  • Disentangled Value: Successor Representations allow the manager to treat bonuses as investments. In hierarchical tasks, the manager learns that paying for a low-reward intermediate step is necessary for a high-reward final outcome.

Results and Generalization

M^3RL consistently approaches the performance of a "Ground Truth" manager that has full access to worker statistics. Notably, the framework exhibits:

  • Rapid Adaptation: The manager quickly adjusts to sudden changes in worker skills mid-training.
  • Zero-Shot Generalization: A manager trained on small teams (4 workers) successfully scales to larger teams (up to 12 workers) and novel environments with different spatial layouts.

Conclusion

M^3RL provides a robust framework for managing self-interested agents in complex, asymmetric environments. While current work focuses on modeling intentions, future research may explore recursive "Theory of Mind" (modeling what workers think of the manager) and addressing "adverse selection," where workers might strategically hide their skills to negotiate better contracts.

Summary updated

BibTeX
@inproceedings{shu2019m,
  author = {Tianmin Shu and Yuandong Tian},
  title = {{M^3RL: Mind-aware Multi-agent Management Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=WL8ICcpaio},
}
Figure