Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis

Hyebin Lee, Seong Tae Kim, Yong Man Ro

Introduction

Deep learning has significantly advanced medical image analysis, yet it often operates as a "black box." In critical fields like oncology, providing a simple classification label such as "Malignant" is insufficient for clinical trust. Clinicians require justifications to understand the reasoning behind an AI's conclusion. This paper introduces a novel framework that generates multimodal justifications—combining visual attention maps with descriptive textual sentences—to explain breast mass diagnoses.

Methodology

The proposed framework consists of two primary components: a Diagnosis Network and a Justification Model.

The overall architecture of the proposed multimodal justification framework, featuring the Diagnosis Network and the Justification Model.

The Diagnosis Network utilizes a VGG16-based encoder to extract visual features from a Region of Interest (ROI) image and predicts the diagnosis. The Justification Model then processes these features and the diagnostic output to generate two types of explanations:

  1. Visual Justification: A spatial attention map (pointing map) that highlights the specific regions of the image influencing the decision.
  2. Textual Justification: A natural language description of the mass's characteristics (e.g., shape and margin).

A key innovation is the Visual Word Constraint ($V_{con}$). Standard language models often suffer from a "duplication problem" in medical contexts, where they repeat generic sentences from the training set. To counter this, the authors implemented a constraint loss ($\mathcal{L}_C$) that ensures the generated "visual words" (e.g., "spiculated," "oval") align with the actual medical features of the input image, forcing the model to provide case-specific descriptions.

Experimental Setup

The model was evaluated using the Digital Database for Screening Mammography (DDSM) and a private Full-Field Digital Mammogram (FFDM) dataset. The researchers curated a dataset where images were annotated with sentences based on the BI-RADS (Breast Imaging Reporting and Data System) lexicon, focusing on shape and margin descriptors to ensure clinical relevance.

Results and Discussion

The inclusion of the visual word constraint loss led to significant improvements in both quantitative metrics and qualitative output.

Qualitative comparison of justifications generated by the proposed method versus a baseline without the visual word constraint loss across various cases.

Quantitatively, the model achieved a BLEU-4 score of 0.0586 and a CIDEr score of 0.1514 on the DDSM dataset, outperforming baseline models. More importantly, the diversity of the generated text improved dramatically: the ratio of novel sentences (those not found in the training set) increased from 4.13% to 43.80%, and the ratio of unique sentences rose to 93.39%. This indicates that the model successfully moved beyond template-matching to provide genuine, image-specific justifications.

Qualitative results (as seen in Figure 5) demonstrate that the proposed method accurately identifies medical features. For instance, in malignant cases with spiculated margins, the model correctly generates descriptions like "sharp lines projecting outside of that mass," whereas the baseline without the constraint loss often produces vague or incorrect descriptors like "too blur to define."

Conclusion

By integrating a visual word constraint, this research addresses the "trust gap" in medical AI. The ability to generate diverse, clinically grounded, and multimodal justifications provides a pathway for more transparent and reliable Computer-Aided Diagnosis systems in clinical practice. Future work will look to expand the lexicon to include more complex descriptors and conduct formal clinical validation with radiologists.

Summary updated

BibTeX
@inproceedings{lee2019generation,
  author = {Hyebin Lee and Seong Tae Kim and Yong Man Ro},
  title = {{Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis}},
  booktitle = {iMIMIC/ML-CDS @ MICCAI},
  year = {2019},
  url = {https://openreview.net/forum?id=e0-oha5i6C5},
}
Figure