TL;DR
This paper tackles the critical challenge of calibrating human trust in AI during collaborative tasks, especially when AI performance is opaque. They propose a framework and two novel self-calibrating trust models (Instance-Based Learning and Reinforcement Learning) that adapt trust dynamically to optimize team performance, outperforming traditional rule-based methods.
What Problem Does This Paper Address?
In human-AI collaboration, optimal team performance hinges on humans trusting AI appropriately. However, this is difficult when humans lack the expertise or transparency to accurately gauge AI performance, leading to potential overtrust (blindly accepting AI errors) or undertrust (missing out on AI benefits). This is particularly problematic with "black-box" AI systems. Existing work often assumes humans can directly observe AI accuracy, which isn't realistic in many high-stakes scenarios. This paper addresses this gap by proposing methods for AI to self-calibrate human trust even when AI performance is uncertain or hidden.
Methods
The authors introduce a theoretical framework modeling trust evolution as a feedback system involving AI's calibrated trust, human judgment, and team performance. They then propose two self-calibrating trust models:
- Instance-Based Trust Calibration (IBLT): This method draws from cognitive science and IBLT. It works by retrieving past experiences (situations, AI's calibrated trust, and resulting team performance) to estimate and dynamically calibrate trust. When faced with a new AI performance situation, it blends information from similar past instances. Human trust is then a linear combination of this calibrated trust and the previous step's team performance.
- Reinforcement Learning-Based Trust Calibration (RL): This approach frames trust calibration as a Markov Decision Process (MDP). The AI learns an optimal policy to adjust its calibrated trust over time to maximize cumulative team performance. They use Q-learning to train a discrete Q-table, where the state is AI performance and the action is the calibrated trust.
Datasets
Not discussed in the paper. The experiments were conducted in a simulated environment.
Key Insights and Ideas
- Dynamic Trust is Crucial: Trust isn't static; it needs to adapt to changing AI performance, especially in dynamic environments.
- Self-Calibration is Key for Opaque AI: When humans can't directly assess AI accuracy, the AI itself needs mechanisms to guide human trust appropriately.
- RL Outperforms for Performance: Reinforcement learning, by directly optimizing for cumulative team performance, shows superior results in adapting trust to maximize outcomes.
- IBLT Offers Interpretability and Efficiency: While not achieving the same peak performance as RL, the Instance-Based method provides a more interpretable approach and is sample-efficient, making it valuable when data is scarce or explainability is paramount.
- Rule-Based Baselines Suffer from Overtrust: Simple heuristic rules for trust adjustment (e.g., increasing trust with better past performance) can lead to persistent overtrust, even when AI performance degrades.
Key Results
In simulations with dynamically changing AI performance (modeled by a decaying sine curve), the RL-based trust calibration consistently outperformed both the IBLT-based method and a rule-based baseline in terms of cumulative team performance. The IBLT method showed stable, albeit more conservative, behavior. The rule-based baseline consistently suffered from overtrust. The paper also shows trust trajectories, illustrating how RL-based trust adapts more dynamically and avoids overtrust compared to the other methods.
Limitations and Future Work
The authors acknowledge that their work is based on simulations. Future work includes conducting user studies to validate these findings in real-world human-AI interaction scenarios. They also suggest exploring hybrid trust calibration models that combine learning-based approaches with explicit social strategies.
Summary updated