Coherent and Concise Radiology Report Generation via Context Specific Image Representations and Orthogonal Sentence States

Litton J. Kurisinkel, Ai Ti Aw, Nancy F. Chen

Introduction

Generating radiology reports from medical imaging is a complex task that extends beyond simple image captioning. Unlike identifying objects in a scene, a radiologist must synthesize multiple views of an X-ray into a structured narrative. This narrative must maintain clinical precision, logical flow, and provide actionable recommendations.

Current "black-box" end-to-end neural models often struggle with this complexity. Common failure modes include hallucinations (reporting non-existent findings), redundancy (repetitive observations), and incoherence (lack of logical transitions between sentences). This research introduces a modular architecture designed to address these issues by providing explicit control over the generation process.

Proposed Architecture

The authors propose a sentence-by-sentence generation framework that incorporates several novel mechanisms to ensure topical continuity and conciseness.

The proposed neural architecture featuring gated content selection and orthogonal state computation.

The architecture, illustrated in Figure 1, centers on three key innovations:

  1. Gated Content Selection: The model employs two distinct gates. The Informative Content (IC) gate filters image features relevant to the current sentence context, while the Topical Continuity (TC) gate ensures that the generated sentence follows logically from the preceding one.
  2. Orthogonal Sentence States (O): To mitigate repetition, the model enforces a diversity constraint. Each new sentence state is mathematically updated to be orthogonal to the average of previous states, forcing the decoder to focus on new information.
  3. Entity-Guided Pre-training (PT): The model is pre-trained to predict medical entities (e.g., "pleural effusion") from images. This provides an inductive bias that grounds the model's internal representations in clinical concepts before it learns full sentence generation.

The system uses a ResNet-152 backbone for image encoding and a Transformer-based encoder for the "indication text," which provides the clinical reason for the imaging.

Experimental Setup

The model was evaluated using the MIMIC-CXR database, a large-scale collection of chest X-rays and associated reports. The researchers utilized a subset of 19,800 entries, extracting 1,060 medical entity clusters for the pre-training task. The data was split into 18,000 records for training, and 900 each for development and testing.

Results and Discussion

The proposed model (Img + IC + TC + O + PT) was compared against standard baselines, including basic RNN and attention-based models.

Quantitative comparison of the proposed model against various experimental settings across text generation and content ordering metrics.

As shown in Figure 2, the full model consistently outperformed baseline configurations. Key findings include:

  • Objective Performance: The model achieved an 18% improvement in informativeness (BLEU/ROUGE) and a 29% improvement in content ordering compared to standard baselines.
  • Conciseness and Flow: The "Orthogonal Sentence States" mechanism effectively reduced redundancy. Qualitative analysis showed the model successfully transitions between different anatomical observations (e.g., lungs to heart to spine) in a logical sequence, whereas baselines often repeated the same lung findings.
  • Human Validation: Blind tests by human evaluators rated the factual accuracy and flow of the proposed method 16% higher than the next-best baseline.

Conclusion

By moving away from purely "black-box" logic and introducing interpretable gates and mathematical constraints for diversity, this architecture significantly improves the quality of automated radiology reports. While the study noted limitations regarding data sparsity for certain medical entities, the results demonstrate that context-specific representations and orthogonality are powerful tools for generating coherent, clinical-grade text. Future work will focus on refining entity clustering and exploring larger datasets to further strengthen the model's medical grounding.

Summary updated

Abstract

Neural models for text generation are often designed in an end-to-end fashion, typically with zero control over intermediate computations, limiting their practical usability in downstream applications. In this work, we incorporate explicit means into neural models to ensure topical continuity, informativeness and content diversity of generated radiology reports. For the purpose we propose a method to compute image representations specific to each sentential context and eliminate redundant content by exploiting diverse sentence states. We conduct experiments to generate radiology reports from medical images of chest x-rays using MIMIC-CXR. Our model outperforms baselines by up to 18% and 29% respective in the evaluation for informativeness and content ordering respectively, relative on objective metrics and 16% on human evaluation.
BibTeX
@inproceedings{kurisinkel2021coherent,
  author = {Litton J. Kurisinkel and Ai Ti Aw and Nancy F. Chen},
  title = {{Coherent and Concise Radiology Report Generation via Context Specific Image Representations and Orthogonal Sentence States}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-industry.31},
}
Figure