Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation

Xingyi Yang, Muchao Ye, Quanzeng You, Fenglong Ma

Introduction

Generating accurate and coherent medical reports from diagnostic images, such as chest X-rays, is a critical yet challenging task in clinical AI. Current methodologies generally fall into two categories: generation-based models, which often produce repetitive and generic text, and retrieval-based models, which rely on static, manually curated templates that struggle to scale or maintain logical flow.

The MedWriter framework addresses these limitations by adopting a "writing by memorizing" paradigm. By mimicking the workflow of radiologists who often adapt existing report templates to new cases, MedWriter utilizes a hierarchical retrieval mechanism to ensure both clinical accuracy and linguistic coherence.

The MedWriter Framework

MedWriter is structured around three core components designed to handle the hierarchical nature of medical documentation: the Visual-Language Retrieval (VLR) module, the Language-Language Retrieval (LLR) module, and a Hierarchical Language Decoder.

Overview of the MedWriter architecture featuring the VLR and LLR modules.

Visual-Language Retrieval (VLR)

The VLR module processes input images to retrieve the most relevant full-report templates from the training dataset. It employs a "pathological attention" mechanism, focusing on specific image regions based on predicted disease labels to ensure the retrieved reports are contextually relevant to the visual findings.

Language-Language Retrieval (LLR)

To maintain logical consistency between sentences, the LLR module retrieves candidate sentences for the subsequent parts of the report based on the previously generated content. This ensures that the report follows a natural progression, avoiding the disjointedness common in standard retrieval models.

Hierarchical Retrieval and Decoding

The generation process is split into two distinct phases: report-level retrieval for the initial sentence and sentence-level retrieval for subsequent entries.

Detailed view of the hierarchical generation process for the first and subsequent sentences.

The Hierarchical Language Decoder serves as the central processing unit, utilizing a two-stage LSTM (Sentence-level and Word-level). It fuses three distinct information streams:

  1. Visual Features: Raw data extracted from the input X-rays.
  2. Report Templates: Global context retrieved by the VLR module.
  3. Sentence Templates: Local context retrieved by the LLR module.

A Multi-query Attention mechanism dynamically weights these inputs, incorporating medical keywords and disease labels to guide the decoder in selecting the most appropriate clinical language.

Experimental Results

The researchers evaluated MedWriter on the Open-I (IU X-Ray) and MIMIC-CXR datasets. The model consistently outperformed state-of-the-art baselines across multiple dimensions:

  • Linguistic Quality: On the large-scale MIMIC-CXR dataset, MedWriter achieved a CIDEr score of 0.306 and a ROUGE-L of 0.332, surpassing previous models like CoAtt.
  • Clinical Accuracy: Using a BERT-based classifier, the model reached an AUC of 0.833 for disease identification, nearing the performance of human-written reports (0.923).
  • Human Evaluation: In a blind study, radiologists rated MedWriter’s reports at 3.68/5.0 for realism, significantly higher than the previous best model (2.50) and close to human-level performance (3.85).

Conclusion

MedWriter demonstrates that "memorizing" and adapting existing clinical language is a powerful strategy for medical report generation. By integrating hierarchical retrieval with a multi-modal decoder, the framework produces reports that are not only linguistically professional but also clinically grounded. Future work will focus on reducing the computational overhead of the retrieval process and incorporating external medical knowledge bases to further refine the model's diagnostic reasoning.

Summary updated

BibTeX
@inproceedings{yang2021writing,
  author = {Xingyi Yang and Muchao Ye and Quanzeng You and Fenglong Ma},
  title = {{Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.387},
}
Figure