Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation

Mingjie Li, Fuyu Wang, Xiaojun Chang, Xiaodan Liang

Abstract

Automated medical report generation is a complex task requiring the synthesis of visual observations into semantically coherent, logically sound medical narratives. Traditional models often struggle with visual noise and data bias, frequently failing to identify rare but critical abnormalities. We present the Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK), a framework that mimics the diagnostic workflow of radiologists by integrating global visual context, fine-grained regions of interest, and external medical textbook knowledge. Evaluated on chest X-ray and COVID-19 CT datasets, ASGK demonstrates superior performance in both quantitative metrics and clinical accuracy.

Introduction

Generating medical reports is significantly more challenging than standard image captioning. Unlike describing a natural scene, a medical report must follow strict medical logic and accurately describe specific pathological findings. Two primary hurdles hinder current AI models:

  1. Visual Noise: Models often fail to distinguish between healthy tissue and the tiny, specific areas (e.g., a small lung shadow) that radiologists focus on.
  2. Data Bias: The prevalence of "normal" cases in medical datasets leads models to over-generate descriptions of healthy organs, often overlooking rare abnormalities.

Methods

The ASGK framework addresses these challenges by incorporating two types of "auxiliary signals" to guide the generation process:

Internal and External Auxiliary Signals

  • Internal Auxiliary Signals (Visual): The model utilizes heatmaps to identify and "crop" regions of primary interest. These fine-grained local features are fused with global image features to ensure subtle abnormalities are captured.
  • External Auxiliary Signals (Linguistic): To bridge the domain gap in medical terminology, the model is pre-trained on large-scale medical textbooks. This provides the encoder with a "prior knowledge" of symptoms and disease correlations.

Architecture

The architecture consists of a Medical Graph Encoder, which represents medical tags as nodes in a graph to learn their correlations, and a Natural Language Decoder inspired by Generative Pre-Training (GPT) to transform graph information into structured paragraphs. To combat data bias, the authors implement Focal Loss, forcing the model to prioritize learning from rare abnormalities over common normal findings.

The ASGK architecture, integrating external auxiliary signals from textbooks and internal visual signals from image regions.

Datasets and Evaluation

The study evaluates ASGK on two key datasets:

  • CX-CHR: A large-scale chest X-ray dataset with over 45,000 images and Chinese diagnostic reports.
  • COV-CTR: A new COVID-19 CT Report dataset containing 728 scans, specifically designed to test robustness in identifying symptoms like "ground-glass opacity."

Results

ASGK outperformed several state-of-the-art baselines, including CoAtt and Vision-BERT, across multiple quantitative and qualitative metrics.

Quantitative comparison of ASGK against baseline models on CX-CHR and COV-CTR datasets.

On the CX-CHR dataset, ASGK achieved a CIDEr score of 324.5, a significant improvement over Vision-BERT's 302.4. Human evaluation by professional radiologists further validated these results, finding ASGK's reports more accurate than baselines in 20% to 27% of cases, while remaining comparable in the majority of other instances.

Conclusion

By mimicking the human expertise of browsing, zooming, and recalling textbook knowledge, ASGK sets a new benchmark for automated medical report generation. Future work will focus on improving the precision of automatic region cropping and expanding the framework to handle a broader variety of medical conditions beyond thoracic imaging.

Summary updated

BibTeX
@article{li2023auxiliary,
  author = {Mingjie Li and Fuyu Wang and Xiaojun Chang and Xiaodan Liang},
  title = {{Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation}},
  journal = {WWW},
  year = {2023},
  doi = {10.1007/s11280-022-01013-6},
}
Figure