Structure Observation Driven Image-Text Contrastive Learning for Computed Tomography Report Generation

Hong Liu, Dong Wei, Qiong Peng, Yawen Huang, Xian Wu, Yefeng Zheng, Liansheng WangIPMI 2025

TL;DR

Researchers have developed a two-stage framework that automates Computed Tomography (CT) report generation by teaching models to "observe" specific anatomical structures. By aligning visual features of organs with their corresponding text descriptions and using a "soft" contrastive learning approach, the model achieves state-of-the-art clinical accuracy while significantly reducing the computational burden of processing massive 3D medical images.

What Problem Does This Paper Address?

Generating medical reports automatically (CTRG) is a "holy grail" for reducing radiologist burnout and improving patient care. While AI has made great strides in 2D X-ray reporting, 3D CT scans present a much steeper challenge.

A single chest X-ray is one image; a chest CT is a 3D volume containing hundreds of slices. Furthermore, while an X-ray might focus on a dozen main findings, a CT scan requires detailed observations of over 80 potential abnormalities across various organs (lungs, heart, liver, etc.). Existing methods often struggle with this "data deluge" or rely on labor-intensive manual labels and rigid knowledge graphs that don't scale well. This paper aims to bridge the gap by creating a more flexible, structure-aware way to link 3D visual data with medical text.

Methods

The authors propose a two-stage framework: Structure Learning and Report Learning.

  1. Stage 1: Structure-Wise Contrastive Learning
    • Visual Queries: Instead of looking at the whole CT scan at once, the model uses "visual queries"—learnable tokens that act like search terms for specific anatomical structures (e.g., "find the lungs," "find the heart").
  2. Textual Alignment: The model extracts sentences from existing reports related to those specific structures. It then uses a contrastive loss to ensure the visual features of an organ match the text describing it.
  3. Soft Pseudo Targets: In standard contrastive learning, the model is punished if it thinks Image A matches Report B. However, in medicine, two different patients might both have "pneumonia." The authors introduced "soft targets" based on text-text similarity, so the model isn't penalized for recognizing identical clinical findings across different cases.
  4. Diversity-Enhanced Queue: To prevent the model from getting "bored" with normal scans, they maintained a queue of diverse, informative text samples to ensure the model learns to distinguish between various rare abnormalities.
  5. Stage 2: Report Generation
    • The visual encoder and queries from Stage 1 are frozen.
  6. Patch Selection: To save memory, the model only selects the top $K$ most relevant image patches for each structure. This reduces the input from thousands of patches to a manageable few hundred.
  7. Text Decoder: A decoder (the authors tested both BERT and LLaMA2) takes these selected patches and generates the final narrative report.

Datasets

The model was evaluated on two public benchmarks:

  • CT-RATE: A large-scale dataset featuring over 25,000 non-contrast chest CT volumes and reports from over 21,000 patients.
  • CTRG-Chest-548K: A smaller dataset of 1,804 chest CT volumes used to test the model's ability to generalize and work with limited data.

Key Insights and Ideas

  • Structure over Words: Most models try to align images with individual words. This paper argues that aligning at the anatomical structure level (e.g., the entire "mediastinum" section) is more natural for radiology and leads to better clinical grounding.
  • Mitigating False Negatives: The "Soft Pseudo Targets" are a clever fix for a common AI problem. By acknowledging that different patients can have the same disease, the model learns more robust medical features rather than just memorizing specific image-report pairs.
  • Efficiency via Selection: By using the learned queries to "crop" the most important parts of the 3D volume, the researchers made it possible to use powerful Large Language Models (LLMs) like LLaMA2 on 3D data that would otherwise exceed hardware memory limits.

Key Results

The framework established new state-of-the-art (SOTA) performance, particularly in Clinical Efficacy (CE) metrics, which measure how accurately the model identifies actual abnormalities.

  • Clinical Accuracy: On the CT-RATE dataset, their BERT-based model achieved an F1 score of 0.310, significantly outperforming previous methods like CT-CLIP (0.145) and PromptMRG (0.288).
  • Transfer Learning: When the model pre-trained on the large CT-RATE dataset was applied to the smaller CTRG-Chest-548K dataset, its performance jumped significantly (F1 score from 0.393 to 0.468), proving that the "structure-learning" stage captures universal medical knowledge.
  • Qualitative Success: The model generated more comprehensive reports that covered multiple organs, whereas previous models often focused narrowly on just the heart or lungs.

Limitations and Future Work

The authors noted that while their model excelled at clinical accuracy (finding the right diseases), its Natural Language Generation (NLG) scores (like BLEU or ROUGE, which measure how much the text "sounds" like the ground truth) were sometimes lower when using LLaMA2 compared to BERT. They attribute this to the difficulty of fine-tuning massive LLMs on relatively small medical datasets.

Future work will likely focus on extending this framework to other types of 3D imaging (like MRIs) and improving the "fluency" of the LLM-generated reports using better fine-tuning techniques.

Summary updated

Abstract

Computed Tomography Report Generation (CTRG) aims to automate the clinical radiology reporting process, thereby reducing the workload of report writing and facilitating patient care. While deep learning approaches have achieved remarkable advances in X-ray report generation, their effectiveness may be limited in CTRG due to larger data volumes of CT images and more intricate details required to describe them. This work introduces a novel two-stage (structure- and report-learning) framework tailored for CTRG featuring effective structure-wise image-text contrasting. In the first stage, a set of learnable structure-specific visual queries observe corresponding structures in a CT image. The resulting observation tokens are contrasted with structure-specific textual features extracted from the accompanying radiology report with a structure-wise image-text contrastive loss. In addition, text-text similarity-based soft pseudo targets are proposed to mitigate the impact of false negatives, i.e., semantically identical image structures and texts from non-paired images and reports. Thus, the model learns structure-level semantic correspondences between CT images and reports. Further, a dynamic, diversity-enhanced negative queue is proposed to guide the network in learning to discriminate various abnormalities. In the second stage, the visual structure queries are frozen and used to select the critical image patch embeddings depicting each anatomical structure, minimizing distractions from irrelevant areas while reducing memory consumption. Also, a text decoder is added and trained for report generation.Our extensive experiments on two public datasets demonstrate that our framework establishes new state-of-the-art performance for CTRG in clinical efficiency, and its components are effective.
BibTeX
@inproceedings{liu2025structure,
  author = {Hong Liu and Dong Wei and Qiong Peng and Yawen Huang and Xian Wu and Yefeng Zheng and Liansheng Wang},
  title = {{Structure Observation Driven Image-Text Contrastive Learning for Computed Tomography Report Generation}},
  booktitle = {IPMI},
  year = {2025},
}
Figure