Do It Like the Doctor: How We Can Design a Model That Uses Domain Knowledge to Diagnose Pneumothorax

Glen Smith, Qiao Zhang, Christopher J. MacLellan

TL;DR

This paper tackles the challenge of diagnosing pneumothorax from ultrasound images, especially with limited data, by incorporating expert physician knowledge. They use a "think-aloud" methodology to extract how doctors identify key features and make diagnoses, then propose using this knowledge to design more robust and explainable AI models.

What Problem Does This Paper Address?

Diagnosing medical conditions from imaging, like pneumothorax from ultrasound, is crucial for patient care. However, building accurate AI models for this task is hindered by limited training data, particularly for rare conditions. This scarcity can lead to poor model performance and over-fitting. The paper aims to bridge this gap by leveraging the invaluable "domain knowledge" of experienced physicians, essentially teaching AI to "think like a doctor" to improve diagnostic accuracy and reliability, especially in time-sensitive scenarios like battlefield medicine.

Methods

The core of this research lies in Cognitive Task Analysis (CTA), specifically the Think-Aloud (TA) method. This involves having medical experts (in this case, two physicians) verbalize their thought processes in real-time as they diagnose pneumothorax from ultrasound videos.

  • Study 1: Knowledge for Making Diagnoses: Focused on eliciting the features and visual characteristics physicians use to make a diagnosis. Participants were shown ultrasound videos and asked to diagnose pneumothorax.
  • Study 2: Knowledge for Explaining Diagnoses: Focused on how physicians explain a diagnosis. Participants were shown videos with either correct or incorrect pre-assigned labels and asked to explain their reasoning. This was designed to see if and how their reasoning changes when presented with potentially misleading information.

The TA sessions were recorded, transcribed, and analyzed to extract keywords, concepts, and decision flows. This information was then used to inform the design of an AI model.

Datasets

  • POCUS ATLAS: Open-source ultrasound video data.
  • Brooke Army Medical Center (BAMC) dataset: Provided ultrasound videos. For this dataset, "sliding" and "no sliding" of the pleural line were used as proxies for "no pneumothorax" and "pneumothorax," respectively.

The study used a total of 12 ultrasound videos (6 for Study 1, 4 for Study 2, with some overlap). The BAMC data provided 32 "sliding" and 30 "no sliding" videos.

Key Insights and Ideas

  • Domain Knowledge is Key for Limited Data: The paper strongly advocates for incorporating expert knowledge as a crucial strategy when dealing with limited datasets in medical AI.
  • "Think Like a Doctor": The TA method effectively extracts the nuanced, step-by-step reasoning process of medical experts, going beyond just identifying features to understanding how they are used.
  • Features vs. Visual Characteristics: The study distinguishes between "features" (e.g., pleural line, rib) and "visual characteristics" (e.g., movement, acoustic shadowing), highlighting that both are critical for diagnosis.
  • Importance of Movement: The "movement" of the pleural line was identified as a particularly critical visual characteristic, with one doctor stating that without it, a diagnosis of pneumothorax would be impossible.
  • Inference Rules for Explainability: Study 2 revealed that experts use "inference rules" – a prioritized sequence of checks – to explain diagnoses, which is vital for building transparent AI systems.
  • Dynamic Features Matter: The use of ultrasound videos allows for the extraction of both static and dynamic features, which is essential for tasks involving motion detection.

Key Results

  • Concept Extraction: The analysis identified twelve key concepts (features and visual characteristics) discussed by the physicians across the two studies. The "pleural line" was consistently identified as a critical feature.
  • Prioritization of Features: The studies revealed a general order in which physicians examine features, with the pleural line being the first point of identification.
  • Inference Rule Identification: Study 2 demonstrated that physicians employ a sequential reasoning process (e.g., check pleural line, then movement, then lung pulse, then vertical artifacts) to arrive at a diagnosis or explanation.
  • Qualitative Agreement: Both physicians generally agreed on diagnoses for videos where the findings were clear, but showed variation and expressed uncertainty when faced with ambiguous cases or incorrect labels.

Limitations and Future Work

  • Limited Sample Size: The studies involved only two physicians, which limits the generalizability of the findings.
  • Proxy Labels: The use of "sliding" and "no sliding" as proxies for pneumothorax diagnosis in the BAMC dataset presented a limitation, as these are not definitive indicators.
  • Future Work:
    • Conduct more think-aloud studies with a larger and more diverse group of physicians to generalize findings.
  • Develop an AI model that incorporates object detection for features and uses inference rules to prioritize them, aiming for reduced training data needs and improved explainability.
  • Extend the approach to diagnose multiple medical conditions from ultrasound videos.
  • Investigate how different orders of utilizing features impact classification accuracy.

Summary updated

Abstract

Computer-aided diagnosis for medical imaging is a well-studied field that aims to provide real-time decision support systems for physicians. These systems attempt to detect and diagnose a plethora of medical conditions across a variety of image diagnostic technologies including ultrasound, x-ray, MRI, and CT. When designing AI models for these systems, we are often limited by little training data, and for rare medical conditions, positive examples are difficult to obtain. These issues often cause models to perform poorly, so we needed a way to design an AI model in light of these limitations. Thus, our approach was to incorporate expert domain knowledge into the design of an AI model. We conducted two qualitative think-aloud studies with doctors trained in the interpretation of lung ultrasound diagnosis to extract relevant domain knowledge for the condition Pneumothorax. We extracted knowledge of key features and procedures used to make a diagnosis. With this knowledge, we employed knowledge engineering concepts to make recommendations for an AI model design to automatically diagnose Pneumothorax.
BibTeX
@inproceedings{smith2022do,
  author = {Glen Smith and Qiao Zhang and Christopher J. MacLellan},
  title = {{Do It Like the Doctor: How We Can Design a Model That Uses Domain Knowledge to Diagnose Pneumothorax}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=D8hBAaQpa9},
}
Figure