Fine-tuning BERT Models for Summarizing German Radiology Findings

Siting Liang, Klaus Kades, Matthias Fink, Peter M. Full, Tim Frederik Weber, Jens Kleesiek, Michael Strube, Klaus Maier‐Hein2022 · DOI

Abstract

Writing the conclusion section of radiology reports is essential for communicating the radiology findings and its assessment to physician in a condensed form. In this work, we employ a transformer-based Seq2Seq model for generating the conclusion section of German radiology reports. The model is initialized with the pretrained parameters of a German BERT model and fine-tuned in our downstream task on our domain data. We proposed two strategies to improve the factual correctness of the model. In the first method, next to the abstractive learning objective, we introduce an extraction learning objective to train the decoder in the model to both generate one summary sequence and extract the key findings from the source input. The second approach is to integrate the pointer mechanism into the transformer-based Seq2Seq model. The pointer network helps the Seq2Seq model to choose between generating tokens from the vocabulary or copying parts from the source input during generation. The results of the automatic and human evaluations show that the enhanced Seq2Seq model is capable of generating human-like radiology conclusions and that the improved models effectively reduce the factual errors in the generations despite the small amount of training data.

Introduction

In clinical oncology, radiology reports are the primary vehicle for communicating findings. While these reports are detailed, physicians often rely on the "Conclusion" section for rapid decision-making. Manually writing these summaries is time-consuming and prone to variability. While Large Language Models (LLMs) are proficient at summarizing general text, medical summarization has a zero-tolerance policy for "hallucinations." A model that generates a fluent sentence but gets a date wrong or misses a new metastasis is clinically dangerous. This paper specifically tackles the challenge of maintaining factual correctness when summarizing structured German radiology reports, where data is often scarce and terminology is highly specialized.

Methods

The researchers developed a hybrid transformer-based model to automatically generate concise "Conclusion" sections from dense German radiology findings. The system is built on a BERT2BERT architecture—a Seq2Seq model where both the encoder and decoder are initialized with pre-trained German BERT weights. This "warm-starting" approach allowed the model to perform better on a relatively small clinical dataset with fewer fine-tuning steps. To move beyond simple abstractive summarization, they implemented two key enhancements:

  1. Extraction Learning Objective (BERT2BERT+Ext): The model is trained with a dual goal. Not only must it generate a summary, but it must also learn to "reconstruct" key sentences from the source findings. To identify these key sentences for training, the authors used a "Longest-k" heuristic, finding that selecting the k longest sentences from the findings section served as an effective proxy for important information.
  2. Pointer Mechanism (BERT2BERT+Ptr): This acts as a "copy-paste" shortcut. At each step of generation, the model decides whether to generate a word from its internal vocabulary or copy a specific token (like a date, a measurement, or a complex anatomical term) directly from the input text.
The BERT2BERT architecture integrated with a pointer mechanism to allow copying from the source text.

The final "hybrid" model (BERT2BERT+Ext+Ptr) combines both strategies to maximize factual grounding.

Datasets

The study utilized a dataset of 10,514 structured oncology reports (SOR) from the University Hospital Heidelberg (UKHD), spanning 2018 and 2019. These reports are organized into sections like "Primary Tumor," "Metastases," and "Reference Measurements." Reports were categorized by patient status: Without Evidence (WE), Partial Response (PR), Stable Disease (SD), and Progressive Disease (PD).

Distribution of samples per class across training, validation, and test sets, highlighting the class imbalance.

The dataset is entirely in German, contributing significantly to non-English clinical NLP. As shown in the figure, there is a notable imbalance, with significantly fewer examples of "Partial Response" and "Progressive Disease" compared to "Stable" or "No Evidence" cases. This imbalance likely limited the model's ability to learn the nuances of deteriorating patient conditions.

Results

The hybrid model (Ext+Ptr) achieved a ROUGE-1 score of 43.32, a significant improvement over the baseline BERT2BERT score of 36.15. In human evaluations, two domain experts (a radiologist and a medical student) rated the summaries on a 0–5 scale. The hybrid models scored near-perfect (close to 5.0) for "Without Evidence" cases and significantly outperformed the baseline in "Oncological Correctness" across all categories.

Qualitative analysis showed that the hybrid models were much more likely to use the correct terminology for tumor assessments (e.g., "Befundkonstanz" vs. "Befundverschlechterung") and effectively eliminated date hallucinations by pulling them directly from the source via the pointer mechanism. However, the model struggled more with "Progressive Disease" reports, where the complexity of findings requires higher-level clinical reasoning.

Discussion and Conclusion

This work demonstrates that combining abstractive summarization with extractive objectives and pointer mechanisms can significantly improve the factual reliability of clinical summaries. While the results are promising, the model's performance was influenced by the complexity of the findings and the inherent data imbalance. Future work should explore domain-adaptive pre-training—training BERT on medical texts before fine-tuning—and test the system on unstructured, free-text radiology notes to assess its broader clinical applicability.

Summary updated

Abstract

Writing the conclusion section of radiology reports is essential for communicating the radiology findings and its assessment to physician in a condensed form. In this work, we employ a transformer-based Seq2Seq model for generating the conclusion section of German radiology reports. The model is initialized with the pretrained parameters of a German BERT model and fine-tuned in our downstream task on our domain data. We proposed two strategies to improve the factual correctness of the model. In the first method, next to the abstractive learning objective, we introduce an extraction learning objective to train the decoder in the model to both generate one summary sequence and extract the key findings from the source input. The second approach is to integrate the pointer mechanism into the transformer-based Seq2Seq model. The pointer network helps the Seq2Seq model to choose between generating tokens from the vocabulary or copying parts from the source input during generation. The results of the automatic and human evaluations show that the enhanced Seq2Seq model is capable of generating human-like radiology conclusions and that the improved models effectively reduce the factual errors in the generations despite the small amount of training data.
BibTeX
@misc{liang2022fine,
  author = {Siting Liang and Klaus Kades and Matthias Fink and Peter M. Full and Tim Frederik Weber and Jens Kleesiek and Michael Strube and Klaus Maier‐Hein},
  title = {{Fine-tuning BERT Models for Summarizing German Radiology Findings}},
  year = {2022},
  doi = {10.18653/v1/2022.clinicalnlp-1.4},
}
Figure