TL;DR
This paper introduces Hierarchical Population Training (HiPT), a novel approach for training AI agents to collaborate effectively with diverse human partners. HiPT uses a hierarchical reinforcement learning structure to learn multiple "best-response" policies, allowing the AI to dynamically adapt its strategy based on the human's play style and skill level, significantly improving collaboration performance and human preference.
What Problem Does This Paper Address?
A major hurdle in human-AI collaboration is the AI's ability to work with novel human partners it hasn't encountered during training. This is particularly challenging because human behavior can be inconsistent, leading to unpredictable action responses. While previous work has shown that training an AI to be a "best response" to a diverse population of trained agents can improve robustness, this doesn't fully address the nuances of human interaction, which involve not just varying skill levels but also diverse play styles and a lack of explicit, consistent strategies. The paper aims to bridge this gap by developing an AI that can robustly collaborate with a wide range of human partners, even those with different skill levels and play styles, without requiring explicit human data for training.
Methods
The core of HiPT is a Hierarchical Reinforcement Learning (HRL) framework adapted for multi-agent collaboration. The AI agent has a two-level structure:
- High-Level Manager Policy: This policy acts as a "manager," observing the environment state and deciding which "sub-policy" to activate.
- Low-Level Sub-Policies: These are designed to be "best-response" policies. The agent learns multiple such sub-policies, each tailored to respond effectively to a specific type of partner. The high-level policy dynamically switches between these low-level policies to adapt to the current human partner.
To train this system, the authors first create a diverse partner population. This population includes agents trained in self-play, with diversity encouraged through a negative Jensen Shannon Divergence (JSD) term in the training objective to promote varied play styles. Skill level diversity is achieved by including agents from different stages of training (fully trained, intermediate checkpoints, and random initializations).
A key innovation is the influence reward added to the high-level policy's objective. This reward encourages the sub-policies to be distinct from each other and from a marginal policy (i.e., the policy without a specific sub-policy prior). This ensures that each learned sub-policy truly represents a different behavioral strategy, making the manager's choice more impactful. The entire system is trained end-to-end using the Proximal Policy Optimization (PPO) algorithm.
Datasets
The primary environment used for training and evaluation is the two-player collaborative Overcooked environment. The authors evaluate their method on five specific layouts of this environment: Cramped Room, Asymmetric Advantages, Coordination Ring, Forced Coordination, and Counter Circuit.
For training the partner population, they use self-play agents. For evaluation, they also use:
- Diverse Self-Play Partners: A population of 16 self-play agents trained with the JSD loss for play-style diversity.
- Human Proxy Partners (Hproxy): A behaviorally cloned human model trained on human-human trajectory data. Crucially, this data is disjoint from any data used for training the AI agents themselves.
- Real Human Partners: A user study was conducted with 72 participants recruited from Amazon Mechanical Turk.
Key Insights and Ideas
- HRL for Adaptive Collaboration: The hierarchical structure is crucial. It allows the AI to learn abstract strategies (which sub-policy to use) and concrete actions, enabling dynamic adaptation to different partners.
- Diversity in Both Play Style and Skill: Simply training against diverse skill levels isn't enough. The inclusion of play-style diversity, enforced by the JSD term, is vital for robust collaboration with a wider range of partners.
- Influence Reward for Distinct Sub-Policies: The influence reward is a clever mechanism to ensure that the learned sub-policies are truly distinct and meaningful, preventing redundancy and improving the manager's decision-making.
- Human Preference Aligns with Performance: The study found that human participants significantly preferred partnering with the HiPT agent, and this preference strongly correlated with the AI's ability to achieve higher rewards. This suggests that HiPT's adaptive capabilities lead to a more satisfying and effective human-AI interaction.
Key Results
- Outperforms Baselines: HiPT consistently outperformed baseline methods (PPOBC and a modified FCP) in terms of average reward when collaborating with both diverse trained agent partners and human proxy partners across all tested Overcooked layouts.
- Influence Reward Benefits Performance: An ablation study showed that the inclusion of the influence reward generally led to higher average rewards, validating its importance in making sub-policies distinct and effective.
- Human Preference: In a user study with real human partners, HiPT was the most preferred AI partner. Participants showed a statistically significant preference for HiPT, with its mean reward being significantly higher than FCP and PPOBC.
Limitations and Future Work
- Environment Specificity: While Overcooked is a complex collaborative environment, the HiPT method's direct applicability to other domains with different interaction dynamics might require further investigation.
- Computational Cost: Training a diverse population and a hierarchical agent can be computationally intensive.
- Future Work: The authors suggest exploring more sophisticated methods for generating diverse partner populations and extending the hierarchical structure to more complex multi-agent scenarios. They also highlight the potential for HiPT to be applied to other human-AI collaboration tasks beyond Overcooked.
Summary updated