Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk Categorization

Ebrahim Rasromani, Stella K. Kang, Yanqi Xu, Beisong Liu, Garvit Luhadia, Wan Fung Chui, Felicia L. Pasadyn, Yu Chih Hung, Julie Y. An, Edwin Mathieu, Zehui Gu, Carlos Fernandez-Granda, Ammar A. Javed, Greg D. Sacks, Tamas Gonda, Chenchan Huang, Yiqiu ShenarXiv 2025 · DOI · arXiv

Introduction

Pancreatic cystic lesions (PCLs) are a common incidental finding, appearing in nearly half of all abdominal MRIs for patients over the age of 60. While the vast majority of these cysts are benign, a small subset can progress to malignancy. Managing this risk requires the meticulous tracking of specific clinical features—such as cyst size, growth rate, and the presence of "mural nodules"—across years of longitudinal radiology reports.

Currently, extracting this data is a massive bottleneck in clinical research. Manual data extraction is labor-intensive and prone to error, while previous automated attempts often relied on expensive closed-source models or lacked the reasoning capabilities to handle complex clinical histories. This paper presents a scalable, interpretable, and cost-effective solution using fine-tuned, open-source Large Language Models (LLMs).

Methodology

The researchers employed a "teacher-student" distillation approach to transfer high-level reasoning capabilities from a large proprietary model to smaller, more efficient open-source models:

  1. Data Generation: GPT-4o acted as the "teacher," processing 6,000 radiology reports. The authors used Chain-of-Thought (CoT) prompting, which forced the model to "show its work" by quoting specific text and explaining its logic before providing a final answer.
  2. Fine-Tuning: Two 8-billion-parameter "student" models—LLaMA-3.1-8B-Instruct and DeepSeek-R1-Distill-Llama-8B—were fine-tuned using QLoRA (Quantized Low-Rank Adaptation). These models were trained specifically on the reasoning traces generated by GPT-4o.
  3. Inference Optimization: To ensure the models could handle high-volume clinical workloads, they were deployed using vLLM, an optimized serving engine that achieved up to a 29x speedup compared to standard decoding methods.
  4. Validation: The system was validated against a "gold standard" of 285 human-annotated reports and a reader study involving three expert radiologists.

Datasets

The study utilized a primary dataset of 6,469 abdominal MRI/CT reports from 5,615 patients, collected at NYU Langone Health between 2005 and 2024. The data was split into 5,153 reports for training, 181 for validation, and 281 for the final test set. A subset of 100 cases was reserved for a head-to-head reader study comparison between the models and human experts.

Key Insights and Results

Performance Parity

The fine-tuned 8B-parameter models achieved 97–98% average exact match accuracy across all clinical features, matching the performance of the much larger GPT-4o. This suggests that specialized fine-tuning can bridge the gap between small open-source models and massive proprietary systems in clinical domains.

Clinical Agreement and Risk Stratification

In the reader study, the models demonstrated "near-perfect" agreement with expert radiologists, achieving a Fleiss’ kappa of 0.893–0.897. Furthermore, the models reached an F1 score of 0.93–0.95 for assigning risk categories (Low Risk vs. Worrisome Features vs. High-Risk Stigmata), which is critical for determining patient management pathways.

The Value of Reasoning

Fine-tuning on reasoning traces (CoT) proved to be a significant performance multiplier. For complex tasks like calculating "Time Intervals" between scans, accuracy improved from 85% to 91% when the model was trained to verbalize the mathematical steps involved.

Economic and Operational Efficiency

While fine-tuning requires an upfront cost (approximately $354 in this study), the variable cost of running open-source models is significantly lower than using proprietary APIs. The authors calculated an "economic break-even" point at roughly 9,100 reports. Additionally, using vLLM allowed the models to process reports in 0.3 to 1.5 seconds, making the processing of million-scale datasets feasible.

Interpretability as a Safety Feature

Unlike traditional "black box" AI, these models generate reasoning traces that quote the original radiology report. This allows clinicians to quickly verify why a model flagged a specific cyst as high-risk. This transparency is essential for clinical safety and for building trust in automated diagnostic aids.

Limitations and Future Work

Despite the promising results, the study notes that "student" models are inherently limited by the quality of the "teacher" model; any biases in GPT-4o could be inherited. Additionally, the models were trained on data from a single institution, and performance may vary with different reporting templates.

Future research will focus on using "Self-Taught Reasoner" (STaR) frameworks to allow models to improve their own reasoning traces iteratively and testing the system across multi-institutional datasets to ensure generalizability.

Summary updated

Abstract

Background: Manual extraction of pancreatic cystic lesion (PCL) features from radiology reports is labor-intensive, limiting large-scale studies needed to advance PCL research. Purpose: To develop and evaluate large language models (LLMs) that automatically extract PCL features from MRI/CT reports and assign risk categories based on guidelines. Materials and Methods: We curated a training dataset of 6,000 abdominal MRI/CT reports (2005-2024) from 5,134 patients that described PCLs. Labels were generated by GPT-4o using chain-of-thought (CoT) prompting to extract PCL and main pancreatic duct features. Two open-source LLMs were fine-tuned using QLoRA on GPT-4o-generated CoT data. Features were mapped to risk categories per institutional guideline based on the 2017 ACR White Paper. Evaluation was performed on 285 held-out human-annotated reports. Model outputs for 100 cases were independently reviewed by three radiologists. Feature extraction was evaluated using exact match accuracy, risk categorization with macro-averaged F1 score, and radiologist-model agreement with Fleiss' Kappa. Results: CoT fine-tuning improved feature extraction accuracy for LLaMA (80% to 97%) and DeepSeek (79% to 98%), matching GPT-4o (97%). Risk categorization F1 scores also improved (LLaMA: 0.95; DeepSeek: 0.94), closely matching GPT-4o (0.97), with no statistically significant differences. Radiologist inter-reader agreement was high (Fleiss' Kappa = 0.888) and showed no statistically significant difference with the addition of DeepSeek-FT-CoT (Fleiss' Kappa = 0.893) or GPT-CoT (Fleiss' Kappa = 0.897), indicating that both models achieved agreement levels on par with radiologists. Conclusion: Fine-tuned open-source LLMs with CoT supervision enable accurate, interpretable, and efficient phenotyping for large-scale PCL research, achieving performance comparable to GPT-4o.
BibTeX
@misc{rasromani2025leveraging,
  author = {Ebrahim Rasromani and Stella K. Kang and Yanqi Xu and Beisong Liu and Garvit Luhadia and Wan Fung Chui and Felicia L. Pasadyn and Yu Chih Hung and Julie Y. An and Edwin Mathieu and Zehui Gu and Carlos Fernandez-Granda and Ammar A. Javed and Greg D. Sacks and Tamas Gonda and Chenchan Huang and Yiqiu Shen},
  title = {{Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk Categorization}},
  year = {2025},
  eprint = {2507.19973},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.19973},
}
Figure