When we design artificial intelligence to assist human experts, we often operate under an idealistic assumption: if we build a highly accurate model and pair it with a human supervisor, the resulting team will perform better than either could alone. This state of mutual optimization is known as complementarity.
In practice, however, human-AI collaboration frequently fails to achieve this synergy. Instead of catching a model's subtle errors, human operators often fall into one of two traps: over-reliance (blindly following an incorrect algorithmic recommendation due to automation bias) or under-reliance (dismissing valid AI insights due to skepticism or algorithm aversion). As Kate Donahue et al. observe in Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness, joint team performance regularly drops below that of the algorithm acting solo because humans lack the meta-knowledge required to know exactly when to override the machine.
To solve this, researchers have converged on the concept of trust calibration: the process of dynamically aligning a human user’s subjective confidence in an AI system with the system's objective, context-specific capabilities. As Alon Jacovi formalizes in Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI, trust is a psychological contract involving vulnerability. If the user's anticipation does not match the model's actual performance, the system inevitably suffers from misuse or disuse.
The Failure of Superficial Explanations
For years, the dominant paradigm in Explainable AI (XAI) assumed that opening the "black box"—by highlighting features or generating saliency maps—would naturally calibrate human trust. Recent empirical work has thoroughly debunked this assumption, revealing that explanations often act as cognitive shortcuts rather than tools for critical thinking.
Samuel Carton et al. demonstrate this friction in Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity. When an AI highlights "toxic" words to justify its classification, it creates a powerful anchoring effect. Humans become significantly faster at completing the task, but they do so by uncritically accepting the AI's framing, failing to notice when the model has fundamentally misinterpreted the context. The explanation increases the plausibility of the output, driving over-reliance rather than calibration.
The missing ingredient in these interfaces is verifiability. Raymond Fok argues in In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making that an explanation only improves team accuracy if it allows a human to quickly and independently confirm the ground truth. If verifying the explanation requires as much cognitive effort as solving the problem from scratch, the human will default to a strategy-graded reliance—trusting the model based on its authoritative tone or its historical average performance rather than evaluating the instance at hand.
Designing for Intuitive Uncertainty
If raw explanations fail to calibrate trust, how can an AI system communicate its boundaries effectively? Standard statistical calibration—ensuring a model's 80% confidence score equates to 80% accuracy—is a mathematical prerequisite, but it often fails to translate to human psychology.
Nina Corvelo Benz demonstrates in Human-Aligned Calibration for AI-Assisted Decision Making that perfectly calibrated models can still lead rational humans to make suboptimal choices if the AI's confidence scores do not scale monotonically with the human's internal certainty. If a model is highly confident in a region where a human expert knows the task is incredibly difficult, the human struggles to form a coherent mental model of when to rely on the system.
To bridge this cognitive gap, researchers are restructuring how uncertainty is presented:
- Frequency Formats: Instead of displaying abstract probabilities (e.g., "Confidence: 71%"), presenting uncertainty as a frequency (e.g., "In 51 out of 72 similar historical cases, this prediction was correct") significantly helps users adjust their reliance and mitigates confirmation bias, as shown by Shiye Cao et al. in Designing for Appropriate Reliance: The Roles of AI Uncertainty Presentation, Initial User Decision, and User Demographics in AI-Assisted Decision-Making.
- Conformal Prediction Sets: Rather than forcing a model to output a single, potentially flawed "best guess," systems can output a variable-sized set of plausible answers that meet a strict mathematical guarantee. Jesse C. Cresswell et al. show in Conformal Prediction Sets Improve Human Decision Making that when a task is difficult, the prediction set naturally grows larger. This expansion acts as an intuitive, non-verbal signal of model uncertainty, prompting the human operator to apply rigorous independent scrutiny exactly when the model is unsure.
The Fragility of the Human Mental Model
Trust calibration is not a static, one-time setup; it is a dynamic relationship that evolves with every interaction. Humans continuously update their internal "generalization function"—their mental map of what the AI can and cannot do. As Keyon Vafa finds in Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function, state-of-the-art models like GPT-4 frequently violate these human expectations, failing unpredictably on tasks that humans assume should be easy based on the model's prior sophisticated answers.
This mental model is incredibly fragile. Kun Yu et al. in Do I trust my machine teammate?: an investigation from perception to decision identify a sharp 70% accuracy threshold: systems operating above this level enjoy growing user reliance, while systems dropping below it trigger a rapid, downward spiral of disengagement and total trust decay.
[AI Performance > 70%] --> Incremental Trust & Reliance
[AI Performance < 70%] --> Rapid Trust Decay & Disuse
This fragility introduces a major paradox when updating AI systems. When a model is upgraded to a version with higher overall accuracy, it often introduces new, unpredictable errors in regions where the old model previously succeeded. Besmira Nushi et al. demonstrate in A Case for Backward Compatibility for Human-AI Teams that this "backward incompatibility" shatters the human's established mental model. The cognitive cost of re-learning the AI's new failure modes can completely outweigh the benefits of the increased algorithmic precision, leading to a net decrease in team performance.
Strategic and Algorithmic Trust Management
Recognizing that human trust fluctuates based on system behavior, researchers have begun designing algorithms that actively manage human trust as a core objective function.
Instead of training models solely to maximize individual accuracy, Besmira Nushi et al. show in Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork that optimizing directly for team utility—penalizing overconfident errors in regions where humans are highly susceptible to bad advice—creates a significantly more effective advisory tool.
Taking this a step further, systems can treat human trust as a hidden variable within a sequential decision-making framework. In Planning with Trust for Human-Robot Collaboration, Stefanos Nikolaidis et al. introduce a Trust-POMDP framework. Under this model, an autonomous agent evaluates the human's trust level in real-time. If the agent senses that the human is beginning to over-trust its capabilities, it may intentionally fail at a low-stakes task. This controlled failure shatters dangerous over-reliance early, recalibrating the human's expectations and ensuring they remain vigilant and ready to intervene during subsequent high-stakes, high-risk scenarios.
By shifting the focus from making AI "perfect" to making AI's boundaries "legible," trust calibration provides the foundation for safe, resilient human-machine partnerships.
Key Foundational Work
Uncertainty Propagation and Trust Building in Visual Analytics (2014) — contributed 2 concept elements to this direction.