Data Extraction from Free-Text Stroke CT Reports Using GPT-4o and Llama-3.3-70B: The Impact of Annotation Guidelines

Jonas Wihl, Enrike Rosenkranz, Severin Schramm, Cornelius Berberich, Michael Griessmair, Piotr Woznicki, Francisco Pinto, Sebastian Ziegelmayer, Lisa C. Adams, Keno K. Bressem, Jan S. Kirschke, Claus Zimmer, Benedikt Wiestler, Dennis M. Hedderich, Su Hwan KimmedRxiv 2025 · DOI

Abstract

To evaluate the performance of LLMs in extracting data from stroke CT reports in the presence and absence of an annotation guideline. Performance of GPT-4o and Llama-3.3-70B in extracting ten imaging findings from stroke CT reports was assessed in two datasets from a single academic stroke center. Dataset A (n = 200) was a stratified cohort including various pathological findings, whereas Dataset B (n = 100) was a consecutive cohort. Initially, an annotation guideline providing clear data extraction instructions was designed based on a review of cases with inter-annotator disagreements in dataset A. For each LLM, data extraction was performed under two conditions – with the annotation guideline included in the prompt and without it. GPT-4o consistently demonstrated superior performance over Llama-3.3-70B. Across both models and both datasets, incorporating the annotation guideline resulted in higher precision rates, while recall rates largely remained stable. Well-defined annotation guidelines can significantly enhance LLM data extraction accuracy in medical reporting.

Introduction

Radiology reports contain critical clinical data but are typically stored as unstructured free-text prose. Extracting this information for research, quality assurance, or registry maintenance currently requires manual labeling by experts, a process that is resource-intensive and prone to variability. While Large Language Models (LLMs) offer a potential automated solution, medical data extraction is complicated by nuances such as chronic vs. acute findings or incidental vs. significant pathologies. This study investigates whether formalizing "rules of the road" through expert-defined annotation guidelines can improve the performance of state-of-the-art LLMs like GPT-4o and Llama-3.3-70B in a high-stakes clinical context.

Methods

The study compared the performance of GPT-4o (proprietary) and Llama-3.3-70B (open-source) in extracting ten specific imaging findings (e.g., hemorrhages, occlusions, aneurysms) into a structured JSON format.

Overview of the study workflow, dataset composition, and LLM data extraction process.

Experimental Design

The core experiment utilized a "before and after" approach:

  1. Base Prompt: Simple instructions to extract findings.
  2. Guideline-Enhanced Prompt: Included a detailed addendum defining "edge cases" (e.g., instructing the AI not to count chronic changes as acute infarcts).

Datasets

Two German-language datasets from a single academic center were used:

  • Dataset A (n=200): A stratified cohort with high pathology frequency, used to identify common confusion points and develop the guidelines.
  • Dataset B (n=100): A consecutive, real-world cohort used as a validation set to test the generalizability of the guidelines.

Results

Precision and Recall

The inclusion of annotation guidelines primarily improved precision (reducing false positives) while maintaining stable recall. In Dataset B, GPT-4o’s precision increased from 0.83 to 0.95, while Llama-3.3-70B improved from 0.87 to 0.94. The most significant gains were observed in variables prone to ambiguity, such as "infarct demarcation" (precision rose from 0.59 to 1.00) and "subdural hematoma" (0.67 to 1.00).

Model Comparison

GPT-4o consistently outperformed Llama-3.3-70B under identical conditions. However, Llama-3.3-70B demonstrated strong potential, achieving precision scores up to 0.86 without guidelines and 0.94 with them. This suggests that high-performing open-source models can be viable for on-premises hospital applications where data privacy is paramount.

Efficiency and Reliability

Both models were significantly faster than human annotators. GPT-4o processed 100 reports in approximately 7.7 minutes, compared to 24 minutes for the local Llama instance and 155 minutes for a human. GPT-4o also showed high reliability (97.6% consistency) across repeated queries, even at higher temperature settings.

Discussion and Conclusion

This study demonstrates that while LLMs are naturally adept at identifying mentions of disease, they require expert-defined rules to accurately categorize clinical significance. The "over-calling" problem—where models flag minor or chronic findings as acute pathologies—can be effectively mitigated through structured prompting.

Limitations include the single-center, single-language nature of the data and the small sample size for rare findings. Future work should explore the extraction of more complex spatial information, such as the specific anatomical location of findings. In conclusion, well-defined annotation guidelines are a critical component for deploying LLMs in automated medical data extraction pipelines.

Summary updated

Abstract

Abstract Purpose To evaluate the performance of LLMs in extracting data from stroke CT reports in the presence and absence of an annotation guideline. Methods In this study, performance of GPT-4o and Llama-3.3-70B in extracting ten imaging findings from stroke CT reports was assessed in two datasets from a single academic stroke center. Dataset A (n = 200) was a stratified cohort including various pathological findings, whereas Dataset B (n = 100) was a consecutive cohort. Initially, an annotation guideline providing clear data extraction instructions was designed based on a review of cases with inter-annotator disagreements in dataset A. For each LLM, data extraction was performed under two conditions – with the annotation guideline included in the prompt and without it. Queries for both LLMs were run with a temperature setting of 0. For GPT-4o, additional queries with a temperature of 1 were performed. Results GPT-4o consistently demonstrated superior performance over Llama-3.3-70B under identical conditions, with micro-averaged precision ranging from 0.83 to 0.95 for GPT-4o and from 0.65 to 0.86 for Llama-3.3-70B. Across both models and both datasets, incorporating the annotation guideline into the LLM input resulted in higher precision rates, while recall rates largely remained stable. In dataset B, precision of GPT-4o and Llama-3-70B improved from 0.83 to 0.95 and from 0.87 to 0.94, respectively. The greatest increase in precision on a variable-level was seen in infarct demarcation (0.59 to 1.00) and subdural hematoma (0.67 to 1.00). Overall classification performance with and without annotation guideline was significantly different in five out of six conditions (e.g. dataset B/Llama-3.3/temp=0: p = 0.001). Conclusion Our results demonstrate the potential of GPT-4o and Llama-3.3-70B in extracting imaging findings from stroke CT reports, with GPT-4o steadily exceeding the performance of Llama-3-70B. We further provide evidence that well-defined annotation guidelines can enhance LLM data extraction accuracy.
BibTeX
@misc{wihl2025data,
  author = {Jonas Wihl and Enrike Rosenkranz and Severin Schramm and Cornelius Berberich and Michael Griessmair and Piotr Woznicki and Francisco Pinto and Sebastian Ziegelmayer and Lisa C. Adams and Keno K. Bressem and Jan S. Kirschke and Claus Zimmer and Benedikt Wiestler and Dennis M. Hedderich and Su Hwan Kim},
  title = {{Data Extraction from Free-Text Stroke CT Reports Using GPT-4o and Llama-3.3-70B: The Impact of Annotation Guidelines}},
  year = {2025},
  doi = {10.1101/2025.01.22.25320938},
  howpublished = {medRxiv},
}
Figure