Multi-agent Imitation Learning with Copulas

Hongwei Wang, Lantao Yu, Zhangjie Cao, Stefano Ermon

Abstract

In multi-agent systems, agent actions are inherently correlated, yet modeling these dependencies efficiently remains a significant challenge. Traditional imitation learning methods often fall into either the "independence trap," resulting in uncoordinated behavior, or the "complexity trap" of opponent modeling, which is computationally expensive and often mathematically inconsistent. We present a novel framework that utilizes copulas to decouple individual agent behaviors from their collective coordination. By leveraging Sklar’s Theorem, we learn marginal action distributions and a copula function separately. This approach achieves state-of-the-art performance in action prediction and trajectory generation across diverse environments, including physics simulations, autonomous driving, and robot soccer, while offering enhanced interpretability and scalability.

Introduction

In multi-agent systems—such as a soccer team or a fleet of autonomous vehicles—agents rarely act in a vacuum. Their actions are deeply correlated. However, most existing imitation learning methods struggle to capture these dependencies. "Mean-field" approaches assume independence, leading to jittery or uncoordinated behavior. Conversely, opponent modeling attempts to capture every agent's view of every other agent, which is computationally redundant and often fails to ensure that learned conditional distributions are mathematically consistent with a single joint policy. We propose a middle ground: a mathematically rigorous, computationally efficient, and scalable method for modeling complex multi-agent dependencies.

Methodology: Decoupling via Copulas

The core innovation of this work is the application of Sklar’s Theorem from probability theory. The theorem states that any multivariate joint distribution can be expressed in terms of its marginals (individual distributions) and a copula (a function describing the dependence structure).

Decomposition of joint distributions into marginals and a copula.

Our framework employs a two-step training procedure:

  1. Learning Marginals: We train a Multi-Layer Perceptron (MLP) to predict the marginal action distribution for each agent. To handle complex, multi-modal behaviors, we utilize a Mixture of Gaussians (MoG).
  2. Learning the Copula: Once individual behaviors are learned, we use a Probability Integral Transform to map actions into a "unit hyper-cube" where the marginals are uniform. We then learn a copula function—either state-dependent via an MLP or state-independent via Kernel Density Estimation (KDE)—to capture the "pure" correlation between agents.
Handling multi-modal coordination using Mixture of Gaussians and copulas.

During inference, the model samples from the copula to obtain a coordinated "seed," which is then transformed back into specific actions for each agent using the inverse of their learned marginals.

Experimental Evaluation

We evaluated our model on three distinct environments:

  • PhySim: A synthetic physics environment where five particles are connected by hidden springs, dictating their coordination.
  • Driving (CARLA): A car-following scenario where a follower car must coordinate its acceleration with a leader car.
  • RoboCup: Real-world robot soccer data involving 10-vs-10 gameplay with distinct roles (defenders, midfielders, forwards).

Key Results and Insights

The copula-based model consistently outperformed baselines like SocialLSTM, Interaction Networks (IN), and Neural Relational Inference (NRI):

  • Action Prediction: Our model achieved the lowest Root Mean Squared Error (RMSE) across all datasets. In the Driving dataset, it reduced error by nearly 35% compared to the next best baseline.
  • Interpretability: Copulas allow for the visualization of agent relationships. In RoboCup experiments, the copula density clearly showed that defenders coordinate strongly with one another but have a much weaker correlation with forwards.
  • Consistency: By building on a joint distribution framework, the model avoids the mathematical inconsistencies that plague models relying on independently learned conditional distributions.

Conclusion and Future Work

By decoupling what an agent does from how it coordinates, our framework enables more flexible and interpretable multi-agent modeling. Future research will address the challenges of decentralized execution, where agents cannot easily sample from a shared copula, and investigate more adaptive copulas for environments where agent roles evolve dynamically.

Summary updated

BibTeX
@inproceedings{wang2021multi,
  author = {Hongwei Wang and Lantao Yu and Zhangjie Cao and Stefano Ermon},
  title = {{Multi-agent Imitation Learning with Copulas}},
  booktitle = {ECML/PKDD},
  year = {2021},
  doi = {10.1007/978-3-030-86486-6\_9},
}
Figure