Leveraging Large Language Models to Extract Prognostic Pathology Features in Ewing Sarcoma

Wenqi Shi, Amin Mehmoodi, Zifan Gu, Zhuo Zhao, Bo Yao, Jennifer O. Black, Jessica Davis, Alyaa al-Ibraheemi, Steven DuBois, Don Barkauskas, Subhash Ramakrishnan, David Hall, Patrick J. Grohar, Yang Xie, Guanghua Xiao, Patrick J. LeaveybioRxiv 2026 · DOI

TL;DR

This paper demonstrates how Large Language Models (LLMs) can effectively extract valuable prognostic information from unstructured pathology reports for Ewing sarcoma, a rare pediatric cancer. By leveraging LLMs, researchers can unlock "dark data" from historical clinical trials to identify novel biomarkers like NSE and S100, which can refine risk stratification and improve patient outcomes.

What Problem Does This Paper Address?

Current risk stratification for Ewing sarcoma primarily relies on clinical factors like metastatic status, often overlooking the prognostic significance of histologic heterogeneity. Pathology reports, rich in biological data, are typically unstructured text, making large-scale retrospective analysis challenging and labor-intensive. This "dark data" represents a missed opportunity for discovering new prognostic indicators. While Large Language Models (LLMs) show promise in extracting clinical data, their application to historical, multi-institutional, scanned pathology reports for rare disease biomarker discovery has not been extensively validated.

Methods

The researchers developed an LLM-based pipeline to extract structured variables from digitized pathology reports. This involved:

  • Optical Character Recognition (OCR): Scanned PDF pathology reports were converted into machine-readable text using Tesseract, an open-source OCR engine. This step was crucial given the historical nature and variable quality of the documents.
  • LLM-Based Feature Extraction: An OpenAI LLM (specifically, the "o3" model) was employed to extract two main categories of data:
  • Immunohistochemical (IHC) Staining Status: The LLM was prompted to classify 17 specific IHC markers (including CD99, NSE, and S100) into "Positive," "Negative," or "Not Specified."
    • CD99 Staining Patterns: The LLM characterized spatial patterns of CD99 staining (e.g., membranous, cytoplasmic, diffuse, focal) based on descriptive text.
  • Ground Truth Validation: A subset of 200 reports was manually annotated by Ewing sarcoma clinical specialists to create a ground truth dataset. This was used to evaluate the LLM's extraction accuracy.
  • Statistical Analysis: Kaplan-Meier analysis and multivariable Cox proportional hazards regression were used to assess the association between extracted features and Overall Survival (OS), adjusting for metastatic status.

Datasets

The study utilized a retrospective cohort of 931 diagnostic pathology reports from six distinct Children's Oncology Group (COG) clinical trials spanning over 21 years. These reports were collected from 185 institutions. A subset of 197 reports was used for ground truth validation, and a further subset of 48 reports was used for inter-rater reliability assessment.

Key Insights and Ideas

  • LLMs as powerful "dark data" excavators: The study highlights the significant potential of LLMs to unlock valuable, previously inaccessible information from unstructured historical medical documents.
  • AI outperforms human annotators: In a cross-validation setting, the LLM achieved a higher accuracy than human experts in extracting IHC marker information, suggesting its resilience to fatigue and errors in low-quality text.
  • Histologic features are potent prognostic indicators: The research demonstrates that specific IHC markers, beyond clinical factors, can significantly refine risk stratification for Ewing sarcoma.
  • NSE as a high-risk marker, S100 as a protective marker: The findings identify Neuron-Specific Enolase (NSE) as a marker associated with worse survival, particularly in localized disease, and S100 as a marker associated with improved survival.

Key Results

  • The LLM achieved a weighted average accuracy of 94% across 17 IHC markers and 98.1% accuracy in a cross-validation subset.
  • The LLM accurately categorized CD99 staining patterns with an estimated accuracy of 90.1%.
  • NSE positivity was independently associated with worse survival (HR = 2.15, p=0.016), with a particularly strong effect in non-metastatic disease (HR = 5.64, p=0.0055).
  • S100 positivity was associated with improved survival (HR = 0.58, p=0.046), with a trend towards significance in non-metastatic disease (HR = 0.44, p=0.088).

Limitations and Future Work

  • OCR quality: The accuracy of LLM extraction is dependent on the quality of the OCR output, which can be affected by noise, typographical errors, and non-English characters.
  • Ambiguous descriptions: Distinguishing between similar CD99 staining patterns (e.g., "Membranous" vs. "Membranous Diffuse") can be challenging due to ambiguous textual descriptions.
  • Multilingual capability: The study notes the opportunity to extract information from non-English narrative reports, suggesting future work in this area.
  • Molecular confirmation: The absence of molecular confirmation of Ewing sarcoma in the dataset is noted as a limitation, as different CD99 staining patterns might suggest non-ETS round cell sarcomas.
  • Future validation: The authors emphasize that these AI-derived histologic insights warrant validation in prospective clinical trials to refine risk stratification for Ewing sarcoma.

Summary updated

Abstract

LLM-assisted extraction of pathology variables is highly accurate and scalable, capable of unlocking "dark data" from historical clinical trials. We identified NSE as a potent risk factor and S100 as a protective marker in Ewing sarcoma, particularly in localized disease. These findings suggest that AI-derived histologic data can refine risk stratification and, if validated, warrant inclusion in future prospective trials.
BibTeX
@misc{shi2026leveraging,
  author = {Wenqi Shi and Amin Mehmoodi and Zifan Gu and Zhuo Zhao and Bo Yao and Jennifer O. Black and Jessica Davis and Alyaa al-Ibraheemi and Steven DuBois and Don Barkauskas and Subhash Ramakrishnan and David Hall and Patrick J. Grohar and Yang Xie and Guanghua Xiao and Patrick J. Leavey},
  title = {{Leveraging Large Language Models to Extract Prognostic Pathology Features in Ewing Sarcoma}},
  year = {2026},
  doi = {10.64898/2026.02.20.707103},
  howpublished = {bioRxiv},
}
Figure