Abstract
Standard automatic evaluation metrics in Question Answering (QA), such as F1 score and Exact Match (EM), often fail to align with human expert preferences, particularly in specialized domains like medicine. This study, conducted during the Kaggle CORD-19 challenge, demonstrates that while models fine-tuned on factoid datasets like SQuAD achieve higher automatic scores by providing brief answers, human users overwhelmingly prefer longer, self-explanatory answers that provide necessary context. We find that incorporating conversational datasets like QuAC improves user satisfaction despite significantly lowering standard performance metrics.
Introduction
The COVID-19 pandemic led to an unprecedented surge in scientific publications, making it nearly impossible for health experts to keep pace. To address this "information overload," researchers developed QA systems to extract specific answers from the CORD-19 corpus. However, a critical challenge emerged: systems optimized for high scores on standard benchmarks were often perceived as less useful by domain experts. This paper investigates the disconnect between character-level overlap metrics and human utility in the context of scientific literature.
Methodology
The researchers implemented a two-stage pipeline to process the CORD-19 dataset, which consists of over 200,000 articles.
Information Retrieval (IR)
The first stage utilizes the BM25F algorithm to search the corpus and retrieve the top 20 most relevant paragraphs for a given query. This module was validated against the TREC-COVID challenge, where it performed competitively among state-of-the-art systems.
Question Answering (QA)
The second stage employs SciBERT, a BERT variant pre-trained on scientific text. Two fine-tuning strategies were compared:
- SQuAD-only: Fine-tuned on SQuAD 2.0, which emphasizes short, fact-based answers.
- SQuAD + QuAC: Further fine-tuned on the Question Answering in Context (QuAC) dataset to encourage more natural, explanatory responses.
The Metric-Preference Disconnect
The core finding of this research is the "Metric Myopia" inherent in standard NLP evaluation. Metrics like F1 and EM measure the overlap between a model's output and a "gold standard" answer. In most benchmarks, gold standards are curated to be the shortest possible span containing the answer.
As shown in the examples above, the SQuAD-only model provides a single-word answer ("nine"), which matches the gold standard well. However, the SQuAD + QuAC model provides a full sentence that explains the context. While this is more helpful to a researcher, the extra words are penalized by the F1 metric, leading to a score of 0.22 versus 0.66.
Results and Analysis
The quantitative results highlight a stark contrast. The SciBERT model trained on SQuAD alone achieved an F1 score of 51.01 on the COVID-QA dataset. When QuAC was added, the F1 score dropped to 40.87.
However, human A/B testing revealed the opposite trend. In blind evaluations, users preferred the QuAC-enhanced model 18.5 times more often than the SQuAD-only model. This preference is largely driven by answer length and completeness.
Figure 2 illustrates that SQuAD-trained models are biased toward extremely short answers (typically under 10 words). In contrast, the COVID-QA expert-annotated answers and the QuAC-influenced model produce longer spans that better match the information-seeking behavior of human experts.
Conclusion
This study serves as a "wake-up call" for the development of QA systems in specialized domains. Optimizing for leaderboards based on F1 and EM can lead to systems that provide technically "correct" but practically useless answers. Future work must focus on developing evaluation metrics that account for naturalness, helpfulness, and the necessity of context in scientific communication.
Summary updated