Know What Not To Know: Users' Perception of Abstaining Classifiers

Andrea Papenmeier, Daniel Hienert, Yvonne Kammerer, Christin Seifert, Dagmar Kern

Abstract

Machine learning systems can help humans to make decisions by providing decision suggestions (i.e., a label for a datapoint). However, individual datapoints do not always provide enough clear evidence to make confident suggestions. Although methods exist that enable systems to identify those datapoints and subsequently abstain from suggesting a label, it remains unclear how users would react to such system behavior. This paper presents first findings from a user study on systems that do or do not abstain from labeling ambiguous datapoints. Our results show that label suggestions on ambiguous datapoints bear a high risk of unconsciously influencing the users' decisions, even toward incorrect ones. Furthermore, participants perceived a system that abstains from labeling uncertain datapoints as equally competent and trustworthy as a system that delivers label suggestions for all datapoints. Consequently, if abstaining does not impair a system's credibility, it can be a useful mechanism to increase decision quality.

Introduction

Decision-Support Systems (DSS) are increasingly integrated into professional and consumer workflows, ranging from medical diagnostics to product selection. Despite their utility, these systems often suffer from a "forced choice" design: they are typically programmed to provide a suggestion for every datapoint, regardless of the quality or ambiguity of the underlying evidence.

This design choice stems from two primary concerns. First, developers fear that admitting uncertainty—having the AI say "I don't know"—will lead users to view the system as incompetent or untrustworthy. Second, humans are susceptible to "automation bias," a tendency to over-rely on automated suggestions even when they are visibly flawed. This study investigates whether an "abstaining" mechanism—where the AI refuses to provide a label for ambiguous data—can mitigate the risks of automation bias without damaging the system's reputation.

Experimental Methodology

The researchers conducted a between-subjects user study with 120 participants. The experimental task involved a text-labeling exercise where users categorized short phrases as either "laptop" or "jacket" (e.g., "a budget one that is durable" vs. "I would need one with lots of memory").

To strictly control the AI's behavior, the team used a simulated (fictive) DSS across three experimental conditions:

  1. C1 (Correct): The AI provided the correct ground-truth label for all phrases, including ambiguous ones.
  2. C2 (Abstain): The AI explicitly stated it could not assign a category for ambiguous phrases.
  3. C3 (Wrong): The AI provided an incorrect label for ambiguous phrases.

For all unambiguous (clear) phrases, the AI consistently provided the correct suggestion. Following the task, participants' performance was measured, and they rated the system’s competence and trustworthiness using standardized psychometric scales.

Data Selection

The study utilized 30 phrases selected from a dataset by Papenmeier et al. (2022). To ensure the distinction between clear and ambiguous data was robust, the authors conducted a preliminary study with 30 crowd workers. The final set included 20 unambiguous phrases (where all workers agreed) and 10 ambiguous phrases (where workers were highly uncertain, such as "quality is most important to me no matter what brand").

Key Findings

The "Invisible" Influence of AI

A significant finding was the power of unconscious influence. While approximately 50% of participants across all conditions claimed they relied primarily on their own judgment rather than the AI, the performance data suggested otherwise. Users in the "Wrong" condition (C3) performed significantly worse than others, demonstrating that AI suggestions exert a powerful pull on human decision-making, even when users believe they are remaining independent.

Trust and Competence in Abstention

Crucially, the study found that honesty does not hurt credibility. There were no statistically significant differences in how users perceived the system across the three conditions. On a 7-point scale, competence was rated between 5.6 and 5.7, and trustworthiness between 5.7 and 5.8 for all groups. This suggests that users do not penalize a system for admitting uncertainty; rather, they seem to accept abstention as a valid and professional response to ambiguity.

Asymmetric Risk and Decision Quality

The study highlighted a stark asymmetry in the risks of AI suggestions on ambiguous data:

  • Correct AI (C1): Resulted in 76% user accuracy.
  • Abstaining AI (C2): Resulted in 69% user accuracy.
  • Wrong AI (C3): Resulted in 42% user accuracy.

The 27% drop in performance caused by a wrong suggestion (C3 vs. C2) far outweighs the modest 7% gain from a correct suggestion on ambiguous data (C1 vs. C2). In high-stakes environments, the "safe" path is for the AI to remain silent rather than risk misguiding the user.

Discussion and Conclusion

While this study utilized a simulated AI and a relatively low-stakes task, the implications for AI design are profound. The results suggest that abstaining classifiers can serve as an effective safeguard for decision quality without compromising user trust.

Future research should explore how these dynamics change in high-stakes fields like medicine or law, and whether providing specific reasons for abstention (such as confidence scores) could further enhance user perception. For now, the evidence suggests that for AI systems, knowing what they do not know—and being honest about it—is a primary mechanism for improving human-AI collaboration.

Summary updated

Abstract

Machine learning systems can help humans to make decisions by providing decision suggestions (i.e., a label for a datapoint). However, individual datapoints do not always provide enough clear evidence to make confident suggestions. Although methods exist that enable systems to identify those datapoints and subsequently abstain from suggesting a label, it remains unclear how users would react to such system behavior. This paper presents first findings from a user study on systems that do or do not abstain from labeling ambiguous datapoints. Our results show that label suggestions on ambiguous datapoints bear a high risk of unconsciously influencing the users' decisions, even toward incorrect ones. Furthermore, participants perceived a system that abstains from labeling uncertain datapoints as equally competent and trustworthy as a system that delivers label suggestions for all datapoints. Consequently, if abstaining does not impair a system's credibility, it can be a useful mechanism to increase decision quality.
BibTeX
@inproceedings{papenmeier2023know,
  author = {Andrea Papenmeier and Daniel Hienert and Yvonne Kammerer and Christin Seifert and Dagmar Kern},
  title = {{Know What Not To Know: Users' Perception of Abstaining Classifiers}},
  booktitle = {Conference on Designing Interactive Systems (Companion Volume)},
  year = {2023},
  url = {https://openreview.net/forum?id=57a7kj2Wxo},
}
Figure