Do it Like the Doctor: How We Can Design a Model That Uses Domain Knowledge to Diagnose Pneumothorax

Glen Smith, Qiao Zhang, Christopher MacLellan

TL;DR

This paper tackles the challenge of designing AI models for medical image diagnosis, particularly for rare conditions like pneumothorax, where limited training data is a major hurdle. The authors propose a novel approach: incorporating expert domain knowledge, elicited through "think-aloud" studies with physicians, to guide the AI model's design, leading to more robust and interpretable diagnostic systems.

What Problem Does This Paper Address?

Developing effective AI models for medical image diagnosis, especially for conditions like pneumothorax, is hampered by a critical issue: a scarcity of training data. This is particularly true for rare diseases, where obtaining sufficient positive examples is difficult. This data limitation often leads to AI models with poor performance, over-fitting to noise, and an inability to generalize well to unseen data. The paper aims to address this by exploring how to leverage the expertise of medical professionals to build more effective AI diagnostic tools, even with limited data.

Methods

The core methodology employed is Cognitive Task Analysis (CTA), specifically the Think-Aloud (TA) method. This involves having medical experts (physicians trained in Point-of-Care Ultrasound or POCUS) verbalize their thought processes while diagnosing pneumothorax from ultrasound videos. The researchers conducted two studies:

  • Study 1: Knowledge for Making Diagnoses: Focused on eliciting the features and visual characteristics that doctors use to make a diagnosis.
  • Study 2: Knowledge for Explaining Diagnoses: Aimed to understand how experts explain a diagnosis, even when presented with potentially incorrect labels, to assess the robustness of their reasoning and identify potential variations in their processes.

The extracted knowledge was then used to inform recommendations for AI model design, emphasizing feature weighting and inference rules.

Datasets

The study utilized two primary sources of ultrasound video data:

  • Brooke Army Medical Center (BAMC) dataset: Provided 32 videos labeled "sliding" (indicating no pneumothorax) and 30 videos labeled "no sliding" (indicating pneumothorax).
  • POCUS ATLAS: An open-source platform providing additional ultrasound videos.

The paper notes that the BAMC dataset's "sliding" and "no sliding" labels were used as proxies for "no pneumothorax" and "pneumothorax," respectively, which is acknowledged as a potential limitation.

Key Insights and Ideas

  • Domain knowledge is crucial for data-scarce AI: The paper strongly advocates for integrating expert knowledge as a fundamental strategy to overcome data limitations in medical AI.
  • Think-Aloud studies are effective for knowledge elicitation: The TA method proved valuable in capturing the nuanced, step-by-step reasoning of physicians, revealing key features and diagnostic procedures.
  • "Pleural line" is a paramount feature: Across both studies, the pleural line was consistently identified as a critical feature for diagnosing pneumothorax.
  • Movement is key to diagnosis: The absence or presence of "movement" (specifically, pleural line movement) was highlighted as a primary indicator, with one doctor stating that diagnosis would be impossible without considering it.
  • Inference rules can guide AI decision-making: Study 2 revealed that experts often employ a sequential, rule-based approach to diagnosis. This suggests that AI models could benefit from similar inference rules to prioritize feature detection and expedite classification.
  • Expert reasoning can be robust to incorrect information: The study showed that experts could critically evaluate provided diagnoses, even when labels were intentionally flipped, demonstrating a strong underlying reasoning process.

Key Results

  • Identification of key features: The studies identified several crucial ultrasound features (e.g., pleural line, B-line, Z-line) and visual characteristics (e.g., lung pulse, movement, vertical artifacts) used by physicians.
  • Prioritization of features: The research suggests a hierarchy of features, with the pleural line being the initial focus, followed by movement, and then other indicators like lung pulse and vertical artifacts.
  • Development of inference rule concepts: The analysis of expert explanations led to the conceptualization of inference rules that mimic the sequential decision-making process of doctors.
  • Qualitative agreement on diagnosis: In Study 1, both experts correctly diagnosed all six videos. In Study 2, while there were instances of disagreement or uncertainty (e.g., regarding lung pulse vs. sliding), the experts' reasoning processes were clearly elucidated.

Limitations and Future Work

  • Limited dataset size: The study acknowledges the use of a relatively small dataset (62 ultrasound videos in Study 1), which inherently limits the generalizability of the findings.
  • Proxy labels: The use of "sliding" and "no sliding" as proxies for pneumothorax diagnosis is noted as a limitation, as sliding is not an entirely definitive indicator.
  • Generalization to other conditions: The current work focuses on pneumothorax. Future work aims to conduct more think-aloud studies to generalize the model to support the diagnosis of multiple medical conditions from ultrasound videos.
  • Model design recommendations: The paper proposes AI model designs incorporating object detection for features and inference rules for sequential analysis, but the actual implementation and evaluation of such models are future steps.
  • Investigating feature order: The authors suggest further investigation into how different orders of utilizing features would contribute to classification accuracy.

Summary updated

Abstract

Computer-aided diagnosis for medical imaging is a well-studied field that aims to provide real-time decision support systems for physicians. These systems attempt to detect and diagnose a plethora of medical conditions across a variety of image diagnostic technologies including ultrasound, x-ray, MRI, and CT. When designing AI models for these systems, we are often limited by little training data, and for rare medical conditions, positive examples are difficult to obtain. These issues often cause models to perform poorly, so we needed a way to design an AI model in light of these limitations. Thus, our approach was to incorporate expert domain knowledge into the design of an AI model. We conducted two qualitative think-aloud studies with doctors trained in the interpretation of lung ultrasound diagnosis to extract relevant domain knowledge for the condition Pneumothorax. We extracted knowledge of key features and procedures used to make a diagnosis. With this knowledge, we employed knowledge engineering concepts to make recommendations for an AI model design to automatically diagnose Pneumothorax.
BibTeX
@inproceedings{smith2022do,
  author = {Glen Smith and Qiao Zhang and Christopher MacLellan},
  title = {{Do it Like the Doctor: How We Can Design a Model That Uses Domain Knowledge to Diagnose Pneumothorax}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=D8hBAaQpa9},
}

Loading related content...

Figure