TL;DR
When AI doesn't know the answer, should it guess or stay silent? This paper explores "abstaining classifiers"—systems that refuse to provide a label for ambiguous data. The study finds that while users are heavily (and often unconsciously) swayed by AI suggestions on tricky data, an AI that admits "I don't know" significantly improves human decision accuracy without damaging the user's trust or the system's perceived competence.
What Problem Does This Paper Address?
Decision-Support Systems (DSS) are everywhere, helping experts diagnose diseases or helping consumers pick products. However, these systems often suffer from a "forced choice" problem: they are typically designed to provide a suggestion for every single datapoint, even when the evidence is murky or contradictory.
The challenge is two-fold. First, humans have a well-documented "automation bias," where they over-rely on AI suggestions, even when they are wrong. Second, there is a fear among developers that if an AI admits it is uncertain, users will view it as incompetent or untrustworthy. This paper investigates whether an "abstaining" mechanism—essentially a "don't know" button for the AI—can prevent users from making errors on ambiguous data without ruining the AI's reputation.
Methods
The researchers conducted a between-subjects user study with 120 participants. The task was a text-labeling exercise where users had to categorize short phrases as either "laptop" or "jacket" (e.g., "a budget one that is durable" vs. "i would need one with lots of memory").
The team used a "fictive" (simulated) DSS to strictly control the AI's behavior across three experimental conditions:
- C1 (Correct): The AI provided the correct ground-truth label even for ambiguous phrases.
- C2 (Abstain): The AI explicitly stated it could not assign a category for ambiguous phrases.
- C3 (Wrong): The AI provided an incorrect label for ambiguous phrases.
For all "unambiguous" (clear) phrases, the AI always provided the correct suggestion. After the task, participants rated the system’s performance, competence, and trustworthiness using standardized scales.
Datasets
The study utilized a dataset of 50 phrases previously published by Papenmeier et al. (2022), categorized as "easy," "difficult," or "impossible" to label.
- Refinement: To ensure the "ambiguous" phrases were truly difficult, the authors ran a preliminary study with 30 crowd workers.
- Final Selection: They narrowed the set to 30 phrases: 20 unambiguous (10 laptop, 10 jacket) where all crowd workers agreed, and 10 ambiguous phrases where crowd workers were highly uncertain (e.g., "quality is most important to me no matter what brand").
Key Insights and Ideas
- The "Invisible" Influence: Even when participants claimed the AI didn't influence them (47-56% of users said they used their own judgment), the data showed otherwise. Users in the "Wrong" condition performed significantly worse, proving that AI suggestions exert a powerful, often unconscious, pull on human decision-making.
- Honesty is the Best Policy: Perhaps the most surprising finding is that abstaining does not hurt credibility. Users rated the "I don't know" system (C2) as just as competent and trustworthy as the system that was "perfect" (C1).
- Asymmetric Risk: The study highlighted that the risk of providing a wrong suggestion on ambiguous data (a 27% drop in user performance) far outweighs the small benefit of getting it right (a 7% gain). In high-stakes environments, the "safe" move is for the AI to remain silent.
Key Results
- User Performance: Accuracy on ambiguous phrases dropped drastically based on the AI's suggestion.
- Correct AI (C1): 76% accuracy
- Abstaining AI (C2): 69% accuracy
- Wrong AI (C3): 42% accuracy
- Perceived Competence & Trust: There were no statistically significant differences in how users perceived the system across the three conditions. On a 7-point scale, competence was rated ~5.6-5.7 and trustworthiness ~5.7-5.8 for all groups.
- Decision Quality: Because the "Abstain" condition didn't lead users toward a wrong choice, it served as an effective safeguard for decision quality.
Limitations and Future Work
The authors acknowledge several limitations:
- Simulated AI: The study used a fictive DSS with pre-programmed "perfect" knowledge of what was ambiguous. In the real world, the AI's detection of its own uncertainty is also prone to error.
- Binary Task: The laptop vs. jacket task is low-stakes. The dynamics of trust and abstention might change in high-stakes fields like medical diagnostics or legal tech.
- Communication Style: The AI simply said it "did not assign a category." Future work could explore if providing reasons for abstaining (e.g., showing a confidence score) further improves user perception.
The researchers conclude that since abstaining doesn't impair a system's credibility, it should be considered a primary design mechanism for dealing with the inherent noise and ambiguity of real-world data.
Summary updated