Introduction
Radiology reports contain critical clinical information but are predominantly stored as unstructured free-text, hindering secondary analysis and longitudinal tracking of medical findings. Researchers from FUJIFILM and Osaka University address this by introducing Finding-Centric Structuring (FCS), a framework designed to convert Japanese radiology reports into structured, finding-specific databases. They also present JRadFCS, the largest multi-facility dataset of its kind, and evaluate the performance of domain-specific models against general-purpose LLMs.
Methods: Finding-Centric Structuring (FCS)
The authors propose a multi-step pipeline to move beyond simple sentence-level extraction to a more clinically relevant finding-centric model.
- NER and RE: A BERT-based model performs Named Entity Recognition (identifying findings, body segments, diagnoses, etc.) and Relation Extraction (linking findings to locations or characteristics).
- Graph Generation: A rule-based processor transforms these entities and relations into "Finding-Centric Graphs." For instance, a single sentence describing multiple locations for a nodule is split into distinct graphs centered on each occurrence.
- Database Conversion: These graphs are finally converted into a structured database format suitable for clinical applications and longitudinal studies.
To support this, the team developed JRadBERT, pre-trained on 758,000 Japanese radiology reports, and introduced the Finding-centric Graph Score (FGS) to measure the accuracy of information "bundles" related to specific findings.
The JRadFCS Dataset
A major contribution of this work is JRadFCS, a large-scale dataset of 8,428 Japanese Computed Tomography (CT) reports.
As shown in the comparison above, JRadFCS is significantly larger and more diverse than previous efforts, incorporating data from seven different facilities. This diversity is crucial for addressing the "generalization gap"—the drop in performance when a model trained on one hospital's data is applied to another due to varying writing styles and terminology.
Key Insights and Performance Analysis
The research highlights several critical findings regarding model generalization and the utility of LLMs in specialized domains:
- Finding-Centric vs. Sentence-Centric: Organizing data around findings better aligns with clinical reasoning and enables precise tracking of medical issues over time.
- The "Out of Entity" (OOE) Factor: The OOE rate—the percentage of medical terms in a new facility's reports not present in the training set—was identified as the primary driver of performance degradation across facilities.
- Strategic Data Augmentation: The authors found that selecting training data based on high OOE or entity length is more effective than random sampling for closing performance gaps between hospitals.
- Domain-Specific BERT vs. GPT-4o: Despite the capabilities of large language models, GPT-4o (20-shot) significantly underperformed compared to JRadBERT. The LLM struggled with strict structural requirements and nuanced clinical relations, scoring 45.26 on the FGS metric compared to JRadBERT's 79.36.
Results
JRadBERT achieved a Macro-F1 score of 79.50 on the FGS across multiple facilities, outperforming other BERT variants like UTH-BERT (42.21) and Tohoku-BERT (77.01). However, a performance gap of approximately 9.5 points remains between the primary training facility and the most challenging external facility, underscoring the ongoing need for multi-facility validation in medical AI.
Conclusion
This work establishes a robust framework for structuring Japanese radiology reports and provides a substantial dataset to advance multi-facility research. While currently limited to Japanese CT reports, future work aims to extend this to other modalities like MRI and Ultrasound and to evaluate the impact of structured data on actual clinical workflows.
Summary updated