Introduction
Open-Domain Question Answering (ODQA) systems, such as Alexa, Siri, and Google Search, have become ubiquitous tools for information retrieval. Despite their sophistication, these models remain imperfect, occasionally providing confident but incorrect answers. To help users navigate these errors, researchers have turned to "explainability"—the practice of providing the evidence or reasoning behind a system's prediction.
This paper identifies and addresses two significant gaps in current explainability research. First, the Modality Gap: while ODQA is frequently used via voice assistants, most evaluation frameworks focus exclusively on visual text displays. Second, the Utility Gap: previous studies have often concluded that natural language explanations do not significantly improve a user's ability to detect errors compared to simply displaying a model's confidence score. This research investigates whether the optimal explanation strategy shifts between spoken and visual interfaces and if explanations can, in fact, improve error-detectability.
Methodology
The authors conducted a large-scale user study involving over 500 participants recruited via Amazon Mechanical Turk. The study utilized a state-of-the-art Dense Passage Retrieval (DPR) model to answer questions from the Natural Questions (NQ) corpus—a dataset of real-world Google Search queries.
Three types of natural language explanations were generated to justify the AI's answers:
- Extractive-Sent: A single, concise sentence from the source Wikipedia article containing the answer.
- Extractive-Long: A full paragraph of context surrounding the answer.
- Abstractive: A human-authored, information-dense summary of the evidence.
Participants evaluated these answers across two modalities: Visual (text on a screen) and Spoken (audio via Text-to-Speech). To ensure high-quality engagement, the researchers implemented an incentive-based payoff matrix. Users were rewarded with a $0.15 bonus for correctly accepting a right answer but penalized $0.15 for accepting a wrong one. This financial stake incentivized participants to critically analyze the provided evidence rather than skimming.
Key Results
The study's primary finding is that the "best" explanation is entirely dependent on the medium of communication. There is no one-size-fits-all approach to AI explainability.
Spoken Modality
In the voice-based interface, brevity was the most critical factor. The Extractive-Sent (short) explanation was the clear winner, achieving 75.6% accuracy in error-detectability. This significantly outperformed the baseline of showing only the model's calibrated confidence score (68.1%). Longer spoken explanations (Extractive-Long) were found to be detrimental, often confusing users or leading them to trust incorrect answers because the lengthy context sounded "plausible."
Visual Modality
Conversely, the visual interface benefited from more detail. The Extractive-Long (paragraph) explanation performed best in this modality, reaching 77.6% accuracy. Because users viewing a screen can scan, skip, and re-read text at their own pace, the additional context provided by a full paragraph served as a helpful verification tool rather than a burden.
Discussion: The Cognitive Bottleneck
The researchers attribute these differences to Cognitive Load. Listening is a linear, time-bound process. According to the Time-Based Resource Sharing (TBRS) model, if a spoken explanation is too long, the user's working memory becomes saturated, causing them to lose track of the original question or the specific answer being justified.
A surprising result was the failure of Abstractive summaries in the spoken modality. While these summaries were short in word count, they were "information-dense." This created a Density Trap: the high concentration of facts made them harder for the brain to process via audio than a simpler, slightly longer extractive sentence. This proves that conciseness in voice UI is about processing ease, not just duration.
Conclusion and Future Work
The research demonstrates that textual explanations are superior to simple confidence scores for fact-based QA, provided they are tailored to the modality. The authors suggest that future AI systems should be modality-aware, automatically truncating evidence for voice assistants while providing richer context for screen-based devices. Furthermore, they propose moving toward interactive explanations, where a system provides a brief initial sentence and allows the user to request more detail if needed, thereby balancing information utility with cognitive load.
Summary updated