Abstract
To evaluate the performance of LLMs in extracting data from stroke CT reports in the presence and absence of an annotation guideline. Performance of GPT-4o and Llama-3.3-70B in extracting ten imaging findings from stroke CT reports was assessed in two datasets from a single academic stroke center. Dataset A (n = 200) was a stratified cohort including various pathological findings, whereas Dataset B (n = 100) was a consecutive cohort. Initially, an annotation guideline providing clear data extraction instructions was designed based on a review of cases with inter-annotator disagreements in dataset A. For each LLM, data extraction was performed under two conditions – with the annotation guideline included in the prompt and without it. GPT-4o consistently demonstrated superior performance over Llama-3.3-70B. Across both models and both datasets, incorporating the annotation guideline resulted in higher precision rates, while recall rates largely remained stable. Well-defined annotation guidelines can significantly enhance LLM data extraction accuracy in medical reporting.
Introduction
Radiology reports contain critical clinical data but are typically stored as unstructured free-text prose. Extracting this information for research, quality assurance, or registry maintenance currently requires manual labeling by experts, a process that is resource-intensive and prone to variability. While Large Language Models (LLMs) offer a potential automated solution, medical data extraction is complicated by nuances such as chronic vs. acute findings or incidental vs. significant pathologies. This study investigates whether formalizing "rules of the road" through expert-defined annotation guidelines can improve the performance of state-of-the-art LLMs like GPT-4o and Llama-3.3-70B in a high-stakes clinical context.
Methods
The study compared the performance of GPT-4o (proprietary) and Llama-3.3-70B (open-source) in extracting ten specific imaging findings (e.g., hemorrhages, occlusions, aneurysms) into a structured JSON format.
Experimental Design
The core experiment utilized a "before and after" approach:
- Base Prompt: Simple instructions to extract findings.
- Guideline-Enhanced Prompt: Included a detailed addendum defining "edge cases" (e.g., instructing the AI not to count chronic changes as acute infarcts).
Datasets
Two German-language datasets from a single academic center were used:
- Dataset A (n=200): A stratified cohort with high pathology frequency, used to identify common confusion points and develop the guidelines.
- Dataset B (n=100): A consecutive, real-world cohort used as a validation set to test the generalizability of the guidelines.
Results
Precision and Recall
The inclusion of annotation guidelines primarily improved precision (reducing false positives) while maintaining stable recall. In Dataset B, GPT-4o’s precision increased from 0.83 to 0.95, while Llama-3.3-70B improved from 0.87 to 0.94. The most significant gains were observed in variables prone to ambiguity, such as "infarct demarcation" (precision rose from 0.59 to 1.00) and "subdural hematoma" (0.67 to 1.00).
Model Comparison
GPT-4o consistently outperformed Llama-3.3-70B under identical conditions. However, Llama-3.3-70B demonstrated strong potential, achieving precision scores up to 0.86 without guidelines and 0.94 with them. This suggests that high-performing open-source models can be viable for on-premises hospital applications where data privacy is paramount.
Efficiency and Reliability
Both models were significantly faster than human annotators. GPT-4o processed 100 reports in approximately 7.7 minutes, compared to 24 minutes for the local Llama instance and 155 minutes for a human. GPT-4o also showed high reliability (97.6% consistency) across repeated queries, even at higher temperature settings.
Discussion and Conclusion
This study demonstrates that while LLMs are naturally adept at identifying mentions of disease, they require expert-defined rules to accurately categorize clinical significance. The "over-calling" problem—where models flag minor or chronic findings as acute pathologies—can be effectively mitigated through structured prompting.
Limitations include the single-center, single-language nature of the data and the small sample size for rare findings. Future work should explore the extraction of more complex spatial information, such as the specific anatomical location of findings. In conclusion, well-defined annotation guidelines are a critical component for deploying LLMs in automated medical data extraction pipelines.
Summary updated