Automatic Fine-grained Segmentation-assisted Report Generation

Frederic Jonske, Constantin Seibold, Osman Alperen Koras, Fin Hendrik Bahnsen, Marie Bauer, Amin Dada, Hamza Kalisch, Anton Schily, Jens KleesiekarXiv 2025 · DOI · arXiv

Introduction

Generating clinical reports from medical images, such as chest X-rays, is a high-stakes task requiring both fluency and precision. Current Large Multimodal Models (LMMs) often struggle with two critical issues: a lack of "grounding"—where findings cannot be traced back to specific visual evidence—and information loss, as general-purpose vision encoders may overlook subtle clinical details. ASaRG (Automatic Segmentation-assisted Report Generation) addresses these challenges by feeding fine-grained anatomical and pathological segmentation maps directly into the model pipeline.

Methods and Architecture

ASaRG is built upon the LLaVA (Large Language-and-Vision Assistant) architecture, which typically maps image features from a vision encoder (ViT) to a Large Language Model (LLM) via a projection layer. ASaRG extends this by incorporating two specialized inputs:

  1. Intermediate Visual Features: Extracted from LVM-Med, a foundation model pre-trained on medical imagery.
  2. Fine-grained Segmentation Maps: 212 distinct masks identifying specific anatomical structures (e.g., individual ribs, heart chambers) and pathological findings (e.g., effusions, foreign objects).
ASaRG Architecture: Integrating fine-grained segmentation and medical visual features into the LLaVA framework.

The researchers found that a simple concatenation strategy for these features was most effective, as it preserves the pre-trained alignment of the base LLM while providing the "local" cues necessary for detailed clinical reporting.

Datasets and Training

The model was evaluated using the MIMIC-CXR dataset, containing over 377,000 chest X-ray images. To align the data with the LLaVA framework, free-text reports were converted into a Visual Question Answering (VQA) format using GPT-3.5. The segmentation models providing the 212 masks were trained on the ChestX-Det and CLiP datasets.

Key Insights

  • Local Cues and Grounding: By providing explicit segmentation masks, ASaRG moves beyond "global" image interpretation. Shuffling experiments confirmed that the model utilizes the semantic meaning of these masks, allowing for grounded reports where specific sentences can be linked to visual regions.
  • Fusion Simplicity: Complex mixing layers can degrade performance by disrupting pre-trained weights. ASaRG demonstrates that adding specialized information as an "appendix" via concatenation is a robust and efficient fusion method.
  • Granularity Trade-offs: While 212 masks were used, grouping them into 20 "superclasses" (e.g., combining all bone segments) yielded nearly identical performance, suggesting a potential for computational optimization without significant loss of detail.

Performance Evaluation

ASaRG achieved a +2.77% improvement in Clinical Efficacy (CE) F1 score over the standard LLaVA baseline. Furthermore, it significantly outperformed existing specialized segmentation-based methods, such as COMG and ORID, by over 6% in F1 score.

Performance comparison of ASaRG variants against baselines and state-of-the-art methods on the MIMIC-CXR dataset.

Notably, these gains were achieved with a minimal increase in parameters (0.06% to 0.43%), demonstrating the high efficiency of the segmentation-assisted approach.

Conclusion and Future Work

ASaRG represents a significant step toward more reliable and interpretable medical report generation. While currently limited to chest X-rays, the architecture is potentially generalizable to other modalities like CT or MRI. Future work will focus on unfreezing model backbones to unlock further performance gains and making the reasoning process between segmentation masks and generated text even more explicit.

Summary updated

Abstract

Reliable end-to-end clinical report generation has been a longstanding goal of medical ML research. The end goal for this process is to alleviate radiologists' workloads and provide second opinions to clinicians or patients. Thus, a necessary prerequisite for report generation models is a strong general performance and some type of innate grounding capability, to convince clinicians or patients of the veracity of the generated reports. In this paper, we present ASaRG (\textbf{A}utomatic \textbf{S}egmentation-\textbf{a}ssisted \textbf{R}eport \textbf{G}eneration), an extension of the popular LLaVA architecture that aims to tackle both of these problems. ASaRG proposes to fuse intermediate features and fine-grained segmentation maps created by specialist radiological models into LLaVA's multi-modal projection layer via simple concatenation. With a small number of added parameters, our approach achieves a +0.89\% performance gain ($p=0.012$) in CE F1 score compared to the LLaVA baseline when using only intermediate features, and +2.77\% performance gain ($p<0.001$) when adding a combination of intermediate features and fine-grained segmentation maps. Compared with COMG and ORID, two other report generation methods that utilize segmentations, the performance gain amounts to 6.98\% and 6.28\% in F1 score, respectively. ASaRG is not mutually exclusive with other changes made to the LLaVA architecture, potentially allowing our method to be combined with other advances in the field. Finally, the use of an arbitrary number of segmentations as part of the input demonstrably allows tracing elements of the report to the corresponding segmentation maps and verifying the groundedness of assessments. Our code will be made publicly available at a later date.
BibTeX
@misc{jonske2025automatic,
  author = {Frederic Jonske and Constantin Seibold and Osman Alperen Koras and Fin Hendrik Bahnsen and Marie Bauer and Amin Dada and Hamza Kalisch and Anton Schily and Jens Kleesiek},
  title = {{Automatic Fine-grained Segmentation-assisted Report Generation}},
  year = {2025},
  eprint = {2507.16623},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.16623},
}
Figure