Introduction
Generating clinical reports from medical images, such as chest X-rays, is a high-stakes task requiring both fluency and precision. Current Large Multimodal Models (LMMs) often struggle with two critical issues: a lack of "grounding"—where findings cannot be traced back to specific visual evidence—and information loss, as general-purpose vision encoders may overlook subtle clinical details. ASaRG (Automatic Segmentation-assisted Report Generation) addresses these challenges by feeding fine-grained anatomical and pathological segmentation maps directly into the model pipeline.
Methods and Architecture
ASaRG is built upon the LLaVA (Large Language-and-Vision Assistant) architecture, which typically maps image features from a vision encoder (ViT) to a Large Language Model (LLM) via a projection layer. ASaRG extends this by incorporating two specialized inputs:
- Intermediate Visual Features: Extracted from LVM-Med, a foundation model pre-trained on medical imagery.
- Fine-grained Segmentation Maps: 212 distinct masks identifying specific anatomical structures (e.g., individual ribs, heart chambers) and pathological findings (e.g., effusions, foreign objects).
The researchers found that a simple concatenation strategy for these features was most effective, as it preserves the pre-trained alignment of the base LLM while providing the "local" cues necessary for detailed clinical reporting.
Datasets and Training
The model was evaluated using the MIMIC-CXR dataset, containing over 377,000 chest X-ray images. To align the data with the LLaVA framework, free-text reports were converted into a Visual Question Answering (VQA) format using GPT-3.5. The segmentation models providing the 212 masks were trained on the ChestX-Det and CLiP datasets.
Key Insights
- Local Cues and Grounding: By providing explicit segmentation masks, ASaRG moves beyond "global" image interpretation. Shuffling experiments confirmed that the model utilizes the semantic meaning of these masks, allowing for grounded reports where specific sentences can be linked to visual regions.
- Fusion Simplicity: Complex mixing layers can degrade performance by disrupting pre-trained weights. ASaRG demonstrates that adding specialized information as an "appendix" via concatenation is a robust and efficient fusion method.
- Granularity Trade-offs: While 212 masks were used, grouping them into 20 "superclasses" (e.g., combining all bone segments) yielded nearly identical performance, suggesting a potential for computational optimization without significant loss of detail.
Performance Evaluation
ASaRG achieved a +2.77% improvement in Clinical Efficacy (CE) F1 score over the standard LLaVA baseline. Furthermore, it significantly outperformed existing specialized segmentation-based methods, such as COMG and ORID, by over 6% in F1 score.
Notably, these gains were achieved with a minimal increase in parameters (0.06% to 0.43%), demonstrating the high efficiency of the segmentation-assisted approach.
Conclusion and Future Work
ASaRG represents a significant step toward more reliable and interpretable medical report generation. While currently limited to chest X-rays, the architecture is potentially generalizable to other modalities like CT or MRI. Future work will focus on unfreezing model backbones to unlock further performance gains and making the reasoning process between segmentation masks and generated text even more explicit.
Summary updated