Towards Visual Dialog for Radiology

Olga Kovaleva, Chaitanya Shivade, Satyananda Kashyap, Karina Kanjaria, Joy T. Wu, Deddeh Ballah, Adam Coy, Alexandros Karargyris, Yufan Guo, David James Beymer, Anna Rumshisky, Vandana MukherjeeBioNLP 2020 · OpenReview

TL;DR

This paper introduces RadVisDial, the first publicly available dataset for visual dialog in radiology, specifically for chest X-rays. They demonstrate that incorporating patient history significantly improves performance over standard VQA models, highlighting the potential of visual dialog as a clinical adjunct.

What Problem Does This Paper Address?

Answering questions about medical images, particularly radiology scans like X-rays, is a crucial task for AI. While Visual Question Answering (VQA) has seen progress, it typically focuses on a single image and question. However, real-world clinical decision-making often involves a dynamic conversation, where a clinician might ask follow-up questions based on initial findings and patient history. This is where Visual Dialog comes in, requiring a system to maintain a coherent conversation about an image, considering previous turns.

The paper identifies a gap in existing research: limited work on intelligent interactive systems for radiology, especially visual dialog. This is problematic because clinicians often rely on radiology reports, which can lead to delays. Visual dialog could potentially expedite patient care by answering preliminary questions without interrupting radiologists, allowing them to focus on more complex interpretations.

Methods

The authors propose a novel task and dataset for visual dialog in radiology. They leverage the MIMIC-CXR dataset, which contains chest X-ray images and associated reports.

  • Dataset Construction: They create a synthetic "silver-standard" dataset by extracting medical conditions as captions and using CheXpert labels to formulate questions and answers. This allows for a large-scale dataset.
  • Model Architectures: They evaluate several state-of-the-art models for visual dialog, including:
  • Stacked Attention Network (SAN): A model that refines question-guided attention over image features iteratively. They modify it to incorporate dialog history.
  • Late Fusion Network (LF): A baseline model that encodes questions, dialog history, and images separately and concatenates them.
  • Recursive Visual Attention (RVA): A model that recursively browses dialog history to resolve co-references.
  • Image Representations: They experiment with different pre-trained image representations, including ResNet-101 and DenseNet-121 features, and a bottom-up attention mechanism.
  • Text Representations: They investigate the impact of different word embeddings (random, GloVe, and domain-specific fastText) and transformer-based contextual vectors (BERT).
  • Contextual Importance: They specifically test the impact of including image captions and dialog history on model performance, comparing VQA-only models with those that incorporate context.

Datasets

  • MIMIC-CXR: The primary source for images and reports.
  • RadVisDial (Synthetic/Silver-Standard): A newly constructed dataset derived from MIMIC-CXR, consisting of 91,060 PA view chest X-ray images with synthetic question-answer pairs based on medical conditions and CheXpert labels. This dataset is made publicly available.
  • Gold Standard Dataset: A smaller dataset of 100 annotated dialogs collected from two expert radiologists conversing about specific chest X-ray images. This was created to compare against the synthetic data and evaluate model performance in a more realistic setting.

Key Insights and Ideas

  • Medical History is Crucial: Incorporating patient medical history (as a caption) and dialog history significantly improves performance compared to models that only consider the image. This is a key differentiator from standard VQA.
  • Domain-Specific Embeddings Matter: Using domain-specific word embeddings (like fastText trained on medical text) leads to better performance than general-purpose embeddings, highlighting the importance of medical knowledge.
  • Data Imbalance is a Major Challenge: The prevalence of "Not in report" answers in the synthetic dataset creates a significant class imbalance, requiring data balancing strategies (downsampling) to prevent biased models.
  • Lateral Views Offer Complementary Information: Including lateral X-ray views alongside frontal views can enhance model performance, suggesting that multi-view information is beneficial.
  • Gold vs. Silver Data Differences: The gold standard data reveals that expert radiologists ask questions with varying granularity and a broader sense (e.g., asking if something is "normal" vs. just if an abnormality is present), which differs from the more focused, CheXpert-derived questions in the silver dataset.

Key Results

  • Baseline Performance: The best-performing baseline model (SAN with full history) achieved a macro-averaged F1-score of 0.34 on the synthetic dataset.
  • Contextual Gains: Adding image captions incrementally improved performance, and incorporating full dialog history led to further gains, particularly in correctly identifying "No" answers.
  • Data Balancing: Downsampling the imbalanced data led to consistent improvements across all models, with SAN showing slightly better performance (0.34 macro F1).
  • Image Representation: ResNet-101 derived image vectors consistently outperformed other tested representations.
  • Lateral View Impact: Appending lateral view vectors to frontal views enhanced SAN's performance, achieving a macro F1 of 0.396 for PA+LAT views.
  • Word Embeddings: Domain-specific fastText embeddings yielded the best results (0.33 macro F1) compared to random or general GloVe embeddings.
  • Gold Standard Fine-tuning: A SAN model pre-trained on silver data and then fine-tuned on the gold standard data achieved the best macro-F1 score of 0.47 on the gold test set.

Limitations and Future Work

  • Limitations:
    • The synthetic dataset is limited to 13 specific abnormalities and 4 answer options, which is a simplification of real-world radiology.
  • The data originates from a single tertiary hospital and a specific patient cohort (Emergency Department, 2012-2014), potentially introducing biases.
  • Chest X-rays are often screening tools, and a diagnosis relies on multiple data points beyond just the image and history. The clinical utility of the current task is therefore limited.
  • Future Work:
    • Diversify questions and answers with expert input and external knowledge bases (e.g., UMLS) to make the synthetic data more realistic.
  • Enrich the data with more question types, such as those addressing the location or size of abnormalities.
  • Collect more real-life dialogs and augment existing datasets to create richer, more expressive dialogs.
  • Develop strategies to manage uncertain labels ('Maybe', 'Not in report') for better data balancing.
  • Bridge the gap between silver and gold data formulations to improve model performance on more natural language interactions.

Summary updated

BibTeX
@inproceedings{kovaleva2020towards,
  author = {Olga Kovaleva and Chaitanya Shivade and Satyananda Kashyap and Karina Kanjaria and Joy T. Wu and Deddeh Ballah and Adam Coy and Alexandros Karargyris and Yufan Guo and David James Beymer and Anna Rumshisky and Vandana Mukherjee},
  title = {{Towards Visual Dialog for Radiology}},
  booktitle = {BioNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=W-WCzOiRYt9},
}
Figure