Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning

Md Saiful Islam, Srijita Das, Sai Krishna Gottipati, William Duguay, Clodéric Mars, Jalal Arabneydi, Antoine Fagette, Matthew Guzdial, Matthew E. TaylorarXiv 2023 · OpenReview

TL;DR

This paper introduces a novel simulator for airport defense where AI-powered drones and human teams collaborate using reinforcement learning. They demonstrate that human-AI collaboration, particularly through policy correction, significantly outperforms fully autonomous AI or human-controlled agents, leading to faster learning and reduced human cognitive load.

What Problem Does This Paper Address?

Protecting critical infrastructure like airports is a complex and expensive task. While AI and reinforcement learning (RL) offer promising solutions, fully autonomous systems have limitations in handling nuanced situations and contextual understanding. Human expertise is valuable, but effectively integrating it into AI systems, especially in dynamic, multi-agent environments, remains a challenge. This paper addresses the question of how well human-AI collaboration performs compared to individual agents or humans alone, and how to best leverage human input for efficient AI learning in such complex scenarios.

Methods

The authors developed a new multi-agent simulator for airport defense, modeling realistic drone dynamics and sensor capabilities. They employed Deep Q-Networks (DQN), specifically D3QN (Dueling Double Deep Q-Network), for training the AI agents. A key contribution is the development of a user interface that allows humans to interact with the AI agents by providing demonstrations and, crucially, by correcting the AI's policy in real-time. They compared different methods of human input: full demonstrations, agent demonstrations, and policy correction.

Datasets

The paper does not explicitly mention pre-existing datasets. Instead, they generated their own datasets through:

  • Agent Demonstrations: Generated by a trained D3QN agent.
  • Human Demonstrations: Collected from 11 human participants using the developed interface.
  • Policy-Corrected Demonstrations: Collected from human participants who corrected the policy of a trained agent.

Key Insights and Ideas

  • Policy Correction is King: The most impactful form of human input for AI learning was policy correction, leading to faster learning and superior performance compared to full demonstrations. This suggests that targeted feedback is more efficient than broad demonstrations.
  • Human-AI Teaming Outperforms All: The combined human-AI approach, especially with policy correction, consistently outperformed both purely AI-driven agents and human-controlled agents in their complex simulation.
  • Reduced Cognitive Load: Human operators experienced significantly lower mental demand, temporal demand, and effort when providing policy corrections compared to when they were fully controlling the drones. This highlights the efficiency gains for humans in collaborative settings.
  • Diversity Matters: While quantity of demonstrations is important, the diversity of strategies and behaviors exhibited in demonstrations (especially policy-corrected ones) appears to be a primary driver of enhanced AI performance.

Key Results

  • Performance Improvement: Human-AI collaboration, particularly with policy correction, achieved a higher success rate (over 80% in the complex scenario) and learned significantly faster (achieving high performance in ~1,000 episodes) compared to D3QN trained from scratch (~5,000+ episodes).
  • Outperforming Baselines: The trained D3QN agent outperformed heuristic-based agents (HA) and average human demonstrators (HM) in both simple and complex scenarios.
  • Efficiency Gains: Policy correction required less human effort and resulted in better performance outcomes than human demonstrations alone.

Limitations and Future Work

  • Simulator Limitations: The authors acknowledge that their simulator, while modeling real-world dynamics, is still a simplification and doesn't fully capture the complexity of real-world scenarios, especially regarding the enemy's capabilities and communication.
  • Focus on Defense: The study is focused on a defensive task, and its findings might not directly translate to offensive or other types of missions.
  • Future Directions: The authors suggest future research should explore more complex scenarios, incorporate diverse human expertise, and investigate how to further enhance human-AI collaboration in real-world settings. They also emphasize the need to address the ethical implications and potential misuse of such technologies.

Summary updated

Abstract

Recent advances in reinforcement learning (RL) and Human-in-the-Loop (HitL) learning have made human-AI collaboration easier for humans to team with AI agents. Leveraging human expertise and experience with AI in intelligent systems can be efficient and beneficial. Still, it is unclear to what extent human-AI collaboration will be successful, and how such teaming performs compared to humans or AI agents only. In this work, we show that learning from humans is effective and that human-AI collaboration outperforms human-controlled and fully autonomous AI agents in a complex simulation environment. In addition, we have developed a new simulator for critical infrastructure protection, focusing on a scenario where AI-powered drones and human teams collaborate to defend an airport against enemy drone attacks. We develop a user interface to allow humans to assist AI agents effectively. We demonstrated that agents learn faster while learning from policy correction compared to learning from humans or agents. Furthermore, human-AI collaboration requires lower mental and temporal demands, reduces human effort, and yields higher performance than if humans directly controlled all agents. In conclusion, we show that humans can provide helpful advice to the RL agents, allowing them to improve learning in a multi-agent setting.
BibTeX
@misc{islam2023human,
  author = {Md Saiful Islam and Srijita Das and Sai Krishna Gottipati and William Duguay and Clodéric Mars and Jalal Arabneydi and Antoine Fagette and Matthew Guzdial and Matthew E. Taylor},
  title = {{Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning}},
  year = {2023},
  eprint = {2312.15160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.15160},
}
Figure