Calibrating AI Trust in Complementary Human-AI Collaboration

Hanjiang Hu, Yifan Sun, Changliu LiuIEEE International Conference on Robotics & Automation Workshop on Public Trust in Autonomous Systems 2025 · OpenReview

TL;DR

This paper tackles the critical challenge of calibrating human trust in AI during collaborative tasks, especially when AI performance is opaque. They propose a framework and two novel self-calibrating trust models (Instance-Based Learning and Reinforcement Learning) that adapt trust dynamically to optimize team performance, outperforming traditional rule-based methods.

What Problem Does This Paper Address?

In human-AI collaboration, optimal team performance hinges on humans trusting AI appropriately. However, this is difficult when humans lack the expertise or transparency to accurately gauge AI performance, leading to potential overtrust (blindly accepting AI errors) or undertrust (missing out on AI benefits). This is particularly problematic with "black-box" AI systems. Existing work often assumes humans can directly observe AI accuracy, which isn't realistic in many high-stakes scenarios. This paper addresses this gap by proposing methods for AI to self-calibrate human trust even when AI performance is uncertain or hidden.

Methods

The authors introduce a theoretical framework modeling trust evolution as a feedback system involving AI's calibrated trust, human judgment, and team performance. They then propose two self-calibrating trust models:

  1. Instance-Based Trust Calibration (IBLT): This method draws from cognitive science and IBLT. It works by retrieving past experiences (situations, AI's calibrated trust, and resulting team performance) to estimate and dynamically calibrate trust. When faced with a new AI performance situation, it blends information from similar past instances. Human trust is then a linear combination of this calibrated trust and the previous step's team performance.
  2. Reinforcement Learning-Based Trust Calibration (RL): This approach frames trust calibration as a Markov Decision Process (MDP). The AI learns an optimal policy to adjust its calibrated trust over time to maximize cumulative team performance. They use Q-learning to train a discrete Q-table, where the state is AI performance and the action is the calibrated trust.

Datasets

Not discussed in the paper. The experiments were conducted in a simulated environment.

Key Insights and Ideas

  • Dynamic Trust is Crucial: Trust isn't static; it needs to adapt to changing AI performance, especially in dynamic environments.
  • Self-Calibration is Key for Opaque AI: When humans can't directly assess AI accuracy, the AI itself needs mechanisms to guide human trust appropriately.
  • RL Outperforms for Performance: Reinforcement learning, by directly optimizing for cumulative team performance, shows superior results in adapting trust to maximize outcomes.
  • IBLT Offers Interpretability and Efficiency: While not achieving the same peak performance as RL, the Instance-Based method provides a more interpretable approach and is sample-efficient, making it valuable when data is scarce or explainability is paramount.
  • Rule-Based Baselines Suffer from Overtrust: Simple heuristic rules for trust adjustment (e.g., increasing trust with better past performance) can lead to persistent overtrust, even when AI performance degrades.

Key Results

In simulations with dynamically changing AI performance (modeled by a decaying sine curve), the RL-based trust calibration consistently outperformed both the IBLT-based method and a rule-based baseline in terms of cumulative team performance. The IBLT method showed stable, albeit more conservative, behavior. The rule-based baseline consistently suffered from overtrust. The paper also shows trust trajectories, illustrating how RL-based trust adapts more dynamically and avoids overtrust compared to the other methods.

Limitations and Future Work

The authors acknowledge that their work is based on simulations. Future work includes conducting user studies to validate these findings in real-world human-AI interaction scenarios. They also suggest exploring hybrid trust calibration models that combine learning-based approaches with explicit social strategies.

Summary updated

Abstract

Human-AI collaboration is a powerful paradigm in decision-making systems, where humans and AI contribute different strengths with clear complementarity. Yet, achieving optimal team performance depends critically on proper trust in AI, ensuring humans rely on AI appropriately. In real-world scenarios, humans often lack the expertise or performance transparency to judge AI accuracy directly, creating a gap in appropriate trust calibration. In this paper, we address this challenge through three key contributions: (1) we propose a theoretical framework modeling the evolution of human trust in AI over time under AI performance uncertainty, (2) we investigate two self-calibrating trust methods, an instance-based cognitive model and a reinforcement learning (RL) model that learns trust calibration policies from experience, and (3) we conduct simulations comparing both approaches against a rule-based baseline under dynamically varying AI performance. Results show that RL-based trust calibration outperforms others in cumulative performance, while instance-based calibration offers interpretability and sample efficiency. These findings offer pathways for safe and adaptive trust alignment in human-AI collaboration toward trustworthy autonomy.
BibTeX
@inproceedings{hu2025calibrating,
  author = {Hanjiang Hu and Yifan Sun and Changliu Liu},
  title = {{Calibrating AI Trust in Complementary Human-AI Collaboration}},
  booktitle = {IEEE International Conference on Robotics \& Automation Workshop on Public Trust in Autonomous Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=6KcewDz76A},
}
Figure