TL;DR
This paper introduces the Bayesian Team Imitation Learner (BTIL), a novel algorithm that learns team policies from suboptimal demonstrations by explicitly modeling team members' time-varying mental states. This approach enables more robust and sample-efficient learning, particularly in scenarios where human-like coordination is crucial and perfect demonstrations are scarce.
What Problem Does This Paper Address?
Teamwork is ubiquitous, from human endeavors to the increasing integration of AI agents into human teams. However, teaching AI agents to effectively collaborate is challenging. Traditional imitation learning (IL) methods learn from expert demonstrations, but real-world teamwork is often suboptimal due to factors like misaligned mental models, imperfect coordination, or varying agent capabilities. Existing multi-agent IL (MAIL) techniques often struggle with these complexities, either assuming optimal behavior, homogeneous agents, or time-invariant latent states. This paper addresses the critical gap of learning team policies from imperfect, suboptimal, and partially observable demonstrations, where team members' internal states (like intentions or preferences) are not directly visible and can change dynamically.
Methods
The core of this work is the Bayesian Team Imitation Learner (BTIL). BTIL tackles the problem by:
- Modeling Latent Mental States: It explicitly models each team member's time-varying "mental state" ($x_i$), which influences their behavior but is not directly observable. This allows BTIL to account for internal motivations and intentions.
- Bayesian Approach: BTIL uses a Bayesian framework to learn from a limited number of demonstrations. This allows it to infer the uncertainty in its learned policies and to be more sample-efficient.
- Semi-Supervised Learning: BTIL is designed to learn from demonstrations where only a subset of the latent states are labeled. This is crucial because manually annotating mental states is labor-intensive.
- Dynamic Bayesian Network (DBN): The paper models team behavior using a DBN, which captures the probabilistic relationships between observable states, actions, and latent mental states, including their temporal dynamics.
- Mean-Field Variational Inference (MFVI): To handle the computational complexity of inferring latent states and policies from partially observed data, BTIL employs MFVI. This technique iteratively updates variational distributions for the policy and latent state dynamics to approximate the true posterior distribution.
Datasets
The paper utilizes two types of datasets:
- Synthetic Datasets: Two synthetic domains, "Movers" and "Cleanup," were created to simulate multi-agent teamwork scenarios with controllable levels of suboptimality and latent state dynamics. These datasets allow for rigorous evaluation against ground truth.
- Human-AI Teamwork Dataset: A novel dataset was collected through a human-subject experiment where participants collaborated with an AI teammate in the Movers and Cleanup domains. This dataset captures more realistic human behavior and coordination challenges.
Key Insights and Ideas
- Suboptimal Demonstrations are Informative: The paper demonstrates that even suboptimal demonstrations, when modeled correctly, can be valuable for learning effective team policies, especially when expert demonstrations are scarce.
- Latent Mental States are Crucial: Explicitly modeling and inferring time-varying latent mental states is key to understanding and replicating complex human-like teamwork, particularly when coordination is critical.
- Semi-Supervision Bridges the Gap: The ability to leverage unlabeled demonstrations significantly improves learning performance, making the approach more practical in real-world scenarios where full labeling is infeasible.
- Joint Learning of Policy and Dynamics: BTIL's ability to jointly infer the team policy and the dynamics of latent states ($T_x$) is essential for robust learning, especially when the latent state transitions are unknown.
Key Results
- Outperforms Baselines: BTIL consistently outperforms baseline methods (Behavioral Cloning, MAGAIL) on synthetic datasets, particularly in terms of policy learning (JS Divergence) and state decoding (Hamming distance), especially in more complex coordination tasks like Movers.
- Effective with Limited Supervision: BTIL-Semi, the semi-supervised version, shows significant performance improvements over fully supervised baselines when only a small fraction of demonstrations are labeled, highlighting its sample efficiency.
- Robust to Unknown Dynamics: BTIL maintains strong performance even when the latent state transition dynamics ($T_x$) are unknown, demonstrating its ability to learn these dynamics alongside the policy.
- Successful on Human-AI Data: BTIL demonstrates promising results on the collected human-AI teamwork dataset, showing its potential for real-world applications.
Limitations and Future Work
- Computational Complexity: The paper notes that approximating the latent state dynamics ($T_x$) can be computationally intensive due to the large state space, requiring approximations.
- Task State Observability: The current formulation assumes the task state ($s$) is fully observable. Future work could extend BTIL to handle partially observable task states.
- Communicative Actions: The paper suggests exploring the integration of communicative actions into the team policy learning framework.
- Role Discovery: While BTIL assumes roles are known, future work could investigate discovering roles within a team.
Summary updated