Do Answers to Boolean Questions Need Explanations? Yes

Sara Rosenthal, Mihaela A. Bornea, Avirup Sil, Radu Florian, J. Scott McCarleyarXiv 2021 · arXiv

Abstract

Current Machine Reading Comprehension (MRC) systems typically provide binary "Yes/No" answers to boolean questions, often leaving users to verify the response against a large block of text. This paper introduces a new dataset of sentence-level evidence-span annotations for the TyDi QA and BoolQ benchmarks. We demonstrate that training models on these granular annotations (BESA) significantly improves span extraction performance and user trust compared to baseline methods.

Introduction

When a user asks a digital assistant a boolean question—for instance, "Can you buy Cadbury's creme eggs all year round?"—a simple "Yes" or "No" is rarely sufficient. Such responses create an "information-hungry" user who must then manually parse the provided context to verify the system's claim. To bridge this gap, explainable AI must provide concise, direct evidence. This research addresses the lack of granular evidence in existing datasets and proposes a method to generate precise justifications for boolean answers.

Dataset and Annotations

The authors contributed a significant new resource: over 3,000 human-annotated evidence spans for two major benchmarks:

  • TyDi QA: A multilingual dataset where boolean questions comprise roughly 10% of the data.
  • BoolQ-X: An expanded version of the BoolQ dataset, where passages were mapped back to their original Wikipedia sources to provide annotators with necessary context.

The annotations go beyond binary labels, marking sentences as "YES," "NO," "WEAK_YES," or "WEAK_NO." This captures the inherent nuance in real-world information, where answers may be inferred or context-dependent. Interestingly, human annotators disagreed with original dataset labels in 5-6% of cases, highlighting the ambiguity often found in "simple" boolean questions.

Methodology

The researchers developed an MRC system using the XLM-Roberta language model and compared three training strategies:

  1. Baseline (BASE): A standard model trained on boolean and short-answer questions, but without evidence spans for the boolean subset.
  2. Boolean Passage Evidence Span (BPES): A model using the entire "gold passage" as a proxy for evidence.
  3. Boolean Evidence Annotated Spans (BESA): The proposed approach, fine-tuned on the new, human-annotated sentence-level evidence spans.

Experimental Results

The BESA model demonstrated superior performance in identifying the exact sentences that justify an answer. As shown in Table 4, BESA achieved a significant improvement in span extraction F1 scores compared to the BASE and BPES models, gaining over 6 F1 points in cross-validation sets.

Table 4: Evidence Span Extraction Results (F1) showing BESA's superior performance across datasets.

Beyond automated metrics, a user study was conducted to evaluate the helpfulness of the system's highlights. Participants ranked model outputs based on relevance and conciseness.

Table 6: Human evaluation results comparing model relevance and user preference.

The results in Table 6 indicate that BESA was the clear favorite among users, achieving a Mean Reciprocal Rank (MRR) of 0.88 for TyDi QA. Users reported that concise, sentence-level highlights significantly increased their confidence in the system's accuracy compared to receiving a full paragraph or no explanation at all.

Discussion and Future Work

The study reveals that "conciseness is king" in user experience; a single, highly relevant sentence is preferred over broader context. However, challenges remain. Approximately 9-13% of questions were flagged as unanswerable based on the provided text, despite having labels in original sources. Furthermore, many questions are "under-specified" (e.g., lacking a time frame). Future work should explore interactive dialogue to clarify these ambiguities and use model confidence scores to provide multi-level highlighting.

Conclusion

Providing explanations for boolean questions is not just a feature; it is a necessity for user trust. By introducing granular evidence annotations and the BESA training strategy, this work provides a path toward more transparent and reliable digital assistants.

Summary updated

Abstract

Existing datasets that contain boolean questions, such as BoolQ and TYDI QA , provide the user with a YES/NO response to the question. However, a one word response is not sufficient for an explainable system. We promote explainability by releasing a new set of annotations marking the evidence in existing TyDi QA and BoolQ datasets. We show that our annotations can be used to train a model that extracts improved evidence spans compared to models that rely on existing resources. We confirm our findings with a user study which shows that our extracted evidence spans enhance the user experience. We also provide further insight into the challenges of answering boolean questions, such as passages containing conflicting YES and NO answers, and varying degrees of relevance of the predicted evidence.
BibTeX
@misc{rosenthal2021do,
  author = {Sara Rosenthal and Mihaela A. Bornea and Avirup Sil and Radu Florian and J. Scott McCarley},
  title = {{Do Answers to Boolean Questions Need Explanations? Yes}},
  year = {2021},
  eprint = {2112.07772},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.07772},
}
Figure