Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis

Hyebin Lee, Seong Tae Kim, Yong Man Ro

Deep learning has revolutionized medical image analysis, but it faces a significant "trust gap." In high-stakes environments like oncology, a "Malignant" label isn't enough; clinicians need to know why the AI reached that conclusion.

In their paper, "Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis," researchers Hyebin Lee, Seong Tae Kim, and Yong Man Ro present a novel framework that doesn't just diagnose breast masses—it explains itself using both visual heatmaps and descriptive sentences.

TL;DR

The researchers developed a deep learning system for breast cancer diagnosis that provides "multimodal justifications" (textual descriptions and visual pointing maps). By introducing a novel visual word constraint loss, the model avoids simply memorizing training reports and instead generates diverse, accurate sentences that correlate directly with the medical features of the image.

What Problem Does This Paper Address?

While Computer-Aided Diagnosis (CADx) systems are highly accurate, they are often "black boxes." Previous attempts at explainability usually provided either a visual heatmap (showing where the AI looked) or a generic text caption.

However, medical text generation suffers from a "duplication problem": because medical datasets are small, models tend to memorize and repeat a few "safe" sentences from the training set rather than describing the specific nuances of a new patient's scan. This lack of specificity makes the AI's "explanation" useless for actual clinical validation.

Methods

The authors propose a two-part architecture consisting of a Diagnosis Network and a Justification Model.

  1. Diagnosis Network: Uses a VGG16-based backbone to extract visual features and predict whether a mass is benign or malignant.
  2. Justification Generator: This is the core innovation. It takes the visual features and the diagnostic decision to produce:
  3. Visual Justification: A spatial attention map (pointing map) highlighting the relevant area.
  4. Textual Justification: A natural language sentence describing the mass (e.g., "A lot of sharp lines overlapped blur the complicated mass").
  5. Visual Word Constraint ($V_{con}$): To solve the duplication problem, the authors added a "visual word constraint loss" ($\mathcal{L}_C$). They pre-trained a sentence classifier to recognize medical descriptors (like "oval" shape or "spiculated" margin) within the generated text. During training, the generator is penalized if the words it chooses don't match the actual medical characteristics of the image.

Datasets

The model was trained and evaluated on two primary sources:

  • DDSM (Digital Database for Screening Mammography): A public dataset of 605 masses.
  • FFDM (Full-Field Digital Mammogram): A private hospital dataset of 147 masses used for cross-validation.
  • BI-RADS Lexicon: The authors curated a custom sentence dataset where each image was annotated with three sentences containing "visual words" corresponding to the standard Breast Imaging Reporting and Data System (BI-RADS) lexicon (e.g., terms for shape and margin).

Key Insights and Ideas

  • Multimodality is Key: Combining a pointing map with text provides a more "persuasive" and verifiable explanation than either alone. The text gives the "what" (descriptors), and the map gives the "where."
  • Fighting Model Laziness: Standard LSTMs are "lazy"—they prefer to output high-probability sequences they've seen before. The Visual Word Constraint forces the model to be "honest" about the specific visual evidence in the current image.
  • Clinical Grounding: By using the BI-RADS lexicon, the model speaks the same language as radiologists, making the output immediately useful for clinical review.

Key Results

The proposed method significantly outperformed baseline models that lacked the constraint loss:

  • Accuracy & Similarity: On the DDSM dataset, the model achieved higher scores across standard NLP metrics: BLEU-4 (0.0586 vs 0.0490) and CIDEr (0.1514 vs 0.1469).
  • Novelty: The most striking result was the reduction in duplication. On the DDSM dataset, the ratio of novel sentences (sentences not found in the training set) jumped from a meager 4.13% to 43.80%.
  • Uniqueness: The ratio of unique sentences generated across the test set rose from 64.46% to 93.39%, proving the model was actually describing individual cases rather than recycling templates.

Limitations and Future Work

  • Dataset Size: As is common in medical AI, the datasets (605 and 147 masses) are relatively small compared to general computer vision datasets.
  • Lexicon Depth: The current model focuses primarily on "shape" and "margin." Future work could expand the lexicon to include more complex descriptors like density or architectural distortion.
  • Clinical Validation: While the metrics show the sentences are more diverse and accurate, the paper does not include a formal "Turing test" or utility study involving human radiologists to see if these justifications actually improve diagnostic speed or accuracy in a real-world setting.

Summary updated

Abstract

The ambiguity of the decision-making process has been pointed out as the main obstacle to practically applying the deep learning-based method in spite of its outstanding performance. Interpretability can guarantee the confidence of the deep learning system, therefore it is particularly important in the medical field. In this study, a novel deep network is proposed to explain the diagnostic decision with visual pointing map and diagnostic sentence justifying result simultaneously. To increase the accuracy of sentence generation, a visual word constraint model is devised in training justification generator. To verify the proposed method, comparative experiments were conducted on the problem of the diagnosis of breast masses. Experimental results demonstrated that the proposed deep network can explain diagnosis more accurately with various textual justifications.
BibTeX
@inproceedings{lee2019generation,
  author = {Hyebin Lee and Seong Tae Kim and Yong Man Ro},
  title = {{Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis}},
  booktitle = {iMIMIC/ML-CDS @ MICCAI},
  year = {2019},
  url = {https://openreview.net/forum?id=e0-oha5i6C5},
}

Loading related content...

Figure