The prevailing dogma in artificial intelligence is that "Human + AI > Human." We assume that by providing an expert with a highly accurate model, the resulting team will naturally inherit the best of both worlds. However, recent research suggests this is often a fallacy. In high-stakes fields like medicine and coding, human-AI teams frequently perform worse than the AI acting alone—a phenomenon known as the complementarity gap.
Achieving true synergy requires moving beyond building "accurate" models and toward engineering systems that explicitly account for human behavior, biases, and existing knowledge.
The Collaboration Paradox
If an AI model is 90% accurate and a human is 80% accurate, we intuitively expect their joint performance to climb toward 95%. Yet, Maia Jacobs and colleagues found that in clinical antidepressant selection, AI recommendations failed to improve clinician accuracy. Even worse, incorrect AI suggestions paired with simple explanations led experts to abandon their own correct judgments How machine-learning recommendations influence clinician treatment selections.
This "collaboration paradox" occurs because humans do not treat AI as a neutral data source. Instead, the interaction is filtered through cognitive bottlenecks:
- Overreliance: Blindly following a model’s incorrect output.
- Underreliance: Dismissing a model’s correct insight because it contradicts intuition. Susanne Gaube et al. argue that in medical imaging, underreliance is actually the more damaging of the two, as experts fail to capitalize on the model's superior pattern recognition Underreliance Harms Human-AI Collaboration More Than Overreliance in Medical Imaging.
- The Paradox of Expertise: Counter-intuitively, highly experienced professionals may be less likely to benefit from AI. Eason Chen observed that experienced tutors often override correct AI suggestions, whereas novices use the AI as a stabilizing anchor AI Knows Best? The Paradox of Expertise, AI-Reliance, and Performance in Educational Tutoring Decision-Making Tasks.
Why Standalone Accuracy is a Misleading Metric
A model that is 99% accurate might be a terrible teammate if its 1% of errors occur exactly where the human also fails. Syed Hasan Amin Mahmood and Ming Yin argue that generalist AI models often duplicate human knowledge, creating high expertise overlap but zero marginal utility Designing Behavior-Aware AI to Improve the Human-AI Team Performance in AI-Assisted Decision Making.
To achieve complementary performance—where the team outperforms its best individual member—the AI must be "behavior-aware." This means training models not to be right all the time, but to be right specifically where the human is likely to be wrong. Bryce McLaughlin and Jann Spiess demonstrate that algorithms designed to anticipate human non-compliance or mistakes lead to significantly better joint outcomes than those designed for maximum standalone precision Designing Algorithmic Recommendations to Achieve Human-AI Complementarity.
Quantifying the Value of Information
If an AI tells a doctor something they already know, it has provided zero "Value of Information" (VoI), regardless of how accurate the statement is. Ziyang Guo, Jessica Hullman, and Jason Hartline have developed decision-theoretic frameworks to measure this "complementary information" The Value of Information in Human-AI Decision-making.
Their method, ILIV-SHAP, shifts the focus of Explainable AI (XAI). Instead of explaining how the model works (which can lead to overreliance), it highlights the features where the AI provides the most unique value relative to the human's baseline knowledge. This helps the human identify when to defer to the machine.
Learning to Defer
A more structural approach to complementarity is the Learning to Defer (L2D) framework. Proposed by David Madras and Richard Zemel, L2D treats the human as a component of the system. The model is trained to predict an outcome or pass the decision to a human expert if it determines the human has a higher probability of being correct Predict Responsibly: Increasing Fairness by Learning To Defer.
This turns the AI into a strategic router. For example, in Human-AI Collaborative Uncertainty Quantification, Sima Noorani and colleagues show that an AI can refine a human's initial prediction set by specifically addressing the human's "blind spots" while avoiding "counterfactual harm"—the act of overriding a human when they were actually right.
From Tools to Co-Reasoning Partners
The future of this field lies in "Collaboration-Necessary" tasks—problems so complex that neither an autonomous agent nor an unaided human can solve them. Hanjun Luo and Chiming Ni demonstrate this in agentic coding: while standalone LLMs and humans both struggle with high-difficulty tasks, their collaboration can nearly double success rates HAI-Eval: Measuring Human-AI Synergy in Collaborative Coding.
By shifting the goal from "model accuracy" to "team complementarity," researchers are moving away from AI as a mere execution tool and toward AI as a strategic partner that understands and compensates for the nuances of human cognition.