Explainable Assessment of Healthcare Articles with QA

Alodie Boissonnet, Marzieh Saeidi, Vassilis Plachouras, Andreas VlachosACL 2022

TL;DR

This paper introduces a novel Question Answering (QA)-based approach for generating explanations for automated healthcare article quality assessments. This method proves competitive with existing summarization-based techniques and offers advantages, particularly when relevant information is missing or for smaller datasets, while also proposing a more appropriate human evaluation protocol for explanation generation.

What Problem Does This Paper Address?

The internet has become a primary source for health information, but it's also rife with low-quality articles containing misinformation and exaggerations, posing a significant threat to public health. While automated systems can classify article quality, their predictions lack trustworthiness without accompanying explanations. Existing methods for generating these explanations, often based on text summarization, struggle with multi-criteria assessments and explaining the absence of information. This paper aims to bridge this gap by developing an explainable quality assessment system that can provide explanations for various quality criteria using a single, unified model.

Methods

The proposed system operates in two main stages:

  1. Article Classification: Articles are first classified according to ten predefined quality criteria. The authors experiment with three models: Logistic Regression, BERT-based classification, and Longformer-based classification. Longformer, a long-document transformer, is favored due to its ability to handle longer texts, which are common in this domain. These models are pre-trained on a biomedical dataset and then fine-tuned on the FakeHealth dataset.
  2. Explanation Generation (QA-based): Instead of summarization, the core innovation is a Question Answering (QA) approach. Each quality criterion is framed as a question, and the model is trained to answer these questions based on the article content and the predicted classification label. This QA objective, implemented using a Longformer encoder-decoder, allows a single model to handle all criteria and classes, leveraging a larger training dataset and better addressing situations where information is missing.

The authors also propose a new human evaluation protocol for assessing explanation quality, focusing on fluency, consistency, and factual correctness, arguing that traditional automatic metrics like ROUGE are insufficient.

Datasets

  • FakeHealth Corpus: This is the primary dataset used for both training and evaluation. It contains health news articles annotated by experts across ten criteria, with associated labels (Satisfactory, Not Satisfactory, Not Applicable) and gold explanations.
  • PubMed: Used for pre-training the classification models.
  • SQuAD v2.0: Used for pre-training the QA-based explanation generation model.

Key Insights and Ideas

  • QA for Explanations is Powerful: Framing explanation generation as a QA task, where criteria become questions, is a significant departure from summarization and proves effective, especially for explaining the absence of information.
  • Complementary Strengths: QA-based and summarization-based explanation systems have complementary strengths. Summarization excels when relevant information is explicitly present, while QA is better when information is missing or for smaller, imbalanced classes. A hybrid approach could yield the best results.
  • Long-Document Models Matter: For classifying and explaining quality in lengthy health articles, models like Longformer that can effectively process long texts significantly outperform those with fixed input length limitations (like BERT).
  • Human Evaluation is Crucial: Automatic metrics like ROUGE correlate poorly with human judgment for evaluating generated explanations, highlighting the need for robust human evaluation protocols that assess fluency, consistency, and factual correctness.

Key Results

  • Classification Performance: Longformer-based models achieved the best macro F₁-scores for quality assessment across criteria, outperforming BERT and Logistic Regression.
  • Explanation Generation: The QA-based approach performed as well as or better than the summarization-based baseline. It produced more fluent explanations and more accurately indicated the correct label.
  • Class-Specific Performance: Summarization models were better for the "Satisfactory" class, while QA models excelled for "Not Satisfactory" and "Not Applicable" classes.
  • Human Evaluation: Pilot studies revealed ambiguities in initial guidelines, leading to a more detailed protocol. The proposed human evaluation protocol is deemed more appropriate than automatic metrics.
  • Correlation with Automatic Metrics: Kendall's Tau correlation coefficients between human judgments and automatic metrics (ROUGE, BLEU) were found to be very low, reinforcing the limitations of automatic evaluation.

Limitations and Future Work

  • Imbalanced Datasets: The FakeHealth dataset exhibits significant class imbalance for some criteria, which affects model performance, particularly for the "Not Applicable" class. While the QA approach helps mitigate this, it remains a challenge.
  • Factual Correctness Stringency: The human evaluation guidelines for factual correctness are very strict, rating explanations as incorrect if even a single detail is wrong, which leads to lower scores for factual correctness even when explanations are generally good.
  • Hybrid Approach: The authors suggest that combining summarization-based systems for "Satisfactory" instances and QA-based systems for others could optimize performance.
  • Further Exploration of QA: The paper opens avenues for exploring QA models more broadly for explainable article assessment.
  • Ethical Considerations: The authors acknowledge that deep learning models can make mistakes and outputs should be treated as recommendations, not ground truth. They also note the environmental benefit of a single QA model versus multiple summarization models.

Summary updated

BibTeX
@inproceedings{boissonnet2022explainable,
  author = {Alodie Boissonnet and Marzieh Saeidi and Vassilis Plachouras and Andreas Vlachos},
  title = {{Explainable Assessment of Healthcare Articles with QA}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure