Improving medical experts' efficiency of misinformation detection: an exploratory study

Aleksandra Nabozny, Bartlomiej Balcerzak, Mikolaj Morzy, Adam Wierzbicki, Pavel Savov, Kamil WarpechowskiWWW 2023 · DOI

Introduction: The Expert Bottleneck in the Infodemic

The rapid spread of medical misinformation online—often termed an "infodemic"—presents a unique challenge for public health. While automated systems can flag suspicious patterns, the nuance required to definitively debunk medical claims necessitates the involvement of highly trained experts. However, medical professionals are a scarce and expensive resource.

The primary obstacle to efficient debunking is the poor "signal-to-noise" ratio in online content. In typical datasets of medical claims, a significant majority of statements are actually credible or neutral. When experts are tasked with manual verification, they spend a disproportionate amount of time reading and approving correct information. This paper introduces a human-in-the-loop framework designed to optimize expert time utilization by re-ranking annotation queues, effectively doubling the efficiency of the debunking process.

Methodology: Topic-Specific Filtering

Rather than attempting to build a single, universal "truth-detector," the researchers developed a framework of topic-specific filtering classifiers. The goal is not to replace the expert, but to sideline highly credible content so that experts focus exclusively on the most suspicious claims.

Technical Components

  • Contextual Units: To address the ambiguity of isolated sentences, the models analyze "triplets" of consecutive sentences, providing the necessary context for accurate classification.
  • High-Precision Filtering: The system is tuned to maximize the recall of the "positive" class (credible/neutral) at a high precision threshold (often >90%). If the model is nearly certain a statement is credible, it is filtered out of the expert's queue.
  • Feature Engineering: The framework utilizes a hybrid approach, combining traditional NLP features (TF-IDF, dependency trees, Named Entity Recognition) with psychological/linguistic features (LIWC) and state-of-the-art embeddings (BioBERT).
  • AutoML Integration: The team employed the TPOT library, which uses genetic algorithms to automatically select the optimal model architecture—ranging from Logistic Regression to XGBoost and Multi-layer Perceptrons—for each specific medical topic.
  • Explainability: To foster trust, the system employs LIME (Local Interpretable Model-agnostic Explanations) to highlight the specific words or features that triggered a classification.

Dataset and Annotation

The study introduced a new, open-sourced dataset comprising over 10,000 sentences extracted from 247 online medical articles. These articles covered ten narrow categories across Cardiology, Pediatry, Gynecology, and Psychiatry.

Table 1: Distribution of expert-annotated labels (Credible, Neutral, Non-credible) across the study's ten medical topics.

Ground truth was established by nine medical experts (MDs and PhDs) with at least a decade of experience. As shown in the data distribution, the volume of credible (CRED) and neutral (NEU) content significantly outweighs non-credible (NONCRED) statements, illustrating the inefficiency of random sampling for expert review.

The Critical Role of Context

A key insight from the research is that credibility markers are highly topic-dependent and context-sensitive. For instance, emotional language might be a red flag in cardiology but common in psychiatry testimonials.

Table 2: Percentage of statements requiring context (m) for credibility assessment. Note that approximately 27% of non-credible statements require at least one sentence of surrounding context (m ≥ 1) to be identified as false.

The study found that over 25% of non-credible statements require surrounding context to be identified as false. This finding justifies the move from sentence-level to triplet-level analysis, ensuring that experts have the necessary information to make a definitive judgment without manually searching for context.

Results and Efficiency Gains

The framework demonstrated a significant impact on expert throughput:

  • Efficiency Boost: On average, the system allowed experts to identify over two times more non-credible statements in the same time interval compared to a random baseline.
  • Filtering Accuracy: The classifiers achieved precision for credible statements ranging from 83.5% to 98.6% across different topics.
  • Expert Throughput: With experts taking between 18 and 35 seconds to evaluate a single statement, the "density" of misinformation provided by the filtering system drastically reduces wasted effort.

Discussion and Future Work

While the system significantly improves efficiency, it carries the risk of false negatives—misinformation that is "clever" enough to be filtered out as credible. However, given that experts cannot possibly review all online content, the authors argue that prioritizing high-probability misinformation is the most effective use of resources.

Future work will focus on integrating automatic topic classifiers to handle mixed-topic streams and exploring the use of this framework as a training tool for medical students to reach expert-level annotation proficiency through iterative feedback.

Summary updated

BibTeX
@article{nabozny2023improving,
  author = {Aleksandra Nabozny and Bartlomiej Balcerzak and Mikolaj Morzy and Adam Wierzbicki and Pavel Savov and Kamil Warpechowski},
  title = {{Improving medical experts' efficiency of misinformation detection: an exploratory study}},
  journal = {WWW},
  year = {2023},
  doi = {10.1007/s11280-022-01084-5},
}
Figure