Visual-Textual Attentive Semantic Consistency for Medical Report Generation

Yi Zhou, Lei Huang, Tao Zhou, Huazhu Fu, Ling Shao

Introduction

Automating the generation of diagnostic reports from medical imaging is a critical yet challenging task in radiology. Unlike general image captioning, medical report generation requires high precision, as missing a tiny abnormality can have severe clinical consequences. Current models often struggle with "semantic consistency"—the alignment between visual evidence in an X-ray and the resulting textual description. Many existing systems fall into the "fluency vs. accuracy" trap, producing grammatically correct prose that fails to accurately reflect the patient's specific condition.

Overview of the medical report generation pipeline, integrating visual features and clinical information to produce accurate diagnostic reports.

Proposed Framework

To address these challenges, this paper introduces a multi-modal framework designed to ensure clinical accuracy through a novel attention mechanism and consistency-driven loss functions.

Multi-Modality Semantic Attention (MMSA)

The core of the architecture is the MMSA module, which synthesizes three distinct data streams:

  1. Regional Visual Features: Extracted from the chest X-ray using a DenseNet-201 backbone.
  2. Predicted Key Findings: The model explicitly predicts disease labels (e.g., "Pleural Effusion") and utilizes their semantic word embeddings as a guide for the language model.
  3. Clinical Information: Patient metadata, such as age, sex, and the clinical "indication" (the reason for the exam), is incorporated to provide necessary diagnostic context.
Detailed architecture of the proposed multi-modal framework, illustrating the interaction between visual encoding, clinical features, and the progressive report decoder.

Progressive Report Decoding

The report generation occurs in two stages. A Sentence Decoder first generates a high-level "topic" embedding for each sentence. Subsequently, a Word Decoder translates these embeddings into specific medical terminology. This hierarchical approach mirrors the structured nature of professional radiology reports.

Consistency Losses

Two novel loss functions are employed to maintain semantic alignment:

  • Image-Sentence Matching Loss ($L_{ISM}$): This ensures that generated sentence embeddings remain mathematically close to the corresponding visual features.
  • Description Accuracy Loss ($L_{DA}$): This penalizes the model for missing specific disease labels or descriptive patterns (e.g., "patchy" or "lower lobe").

Experimental Setup and Results

The framework was evaluated on the MIMIC-CXR and IU X-Ray datasets, the two primary benchmarks in the field.

Performance Metrics

Beyond standard linguistic metrics like BLEU and CIDEr, the authors introduce nKTD (normalized Key Term Distance). This metric specifically measures the accuracy of clinical "key terms," providing a more meaningful assessment of diagnostic truth than word-overlap alone.

Key Results

The proposed model achieved state-of-the-art (SOTA) performance across nearly all metrics. Notably, on the MIMIC-CXR dataset, it reached an nKTD score of 0.106, a significant improvement over the previous best of 0.154. Ablation studies confirmed that both the MMSA module and the consistency losses are vital to the model's success, particularly in reducing "hallucinations" and improving the detection of subtle findings.

Conclusion

By treating clinical context as a necessary prior and using disease labels as a semantic bridge, this framework significantly advances the state of automated medical reporting. Future work will focus on refining label accuracy and extending the model to multi-view reasoning, integrating both frontal and lateral X-ray views to further emulate a radiologist's workflow.

Summary updated

BibTeX
@inproceedings{zhou2021visual,
  author = {Yi Zhou and Lei Huang and Tao Zhou and Huazhu Fu and Ling Shao},
  title = {{Visual-Textual Attentive Semantic Consistency for Medical Report Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00395},
}
Figure