This paper presents an end-to-end agentic workflow for automatically generating clinical data schemas and extracting information from medical literature. By applying this to a large corpus of sarcoma case report abstracts, the authors demonstrate that large language models (LLMs) can move beyond simple data extraction to "bottom-up" schema design, though human expert curation remains a critical final step.
TL;DR
The authors developed an LLM-based agentic pipeline that automatically induces a Common Data Element (CDE) schema from ~25,000 sarcoma abstracts and uses it to extract structured data with 94.9% accuracy. While the extraction is highly reliable, the study highlights that agent-generated schemas still require human expert review to resolve redundancies and conceptual errors.
What Problem Does This Paper Address?
In clinical research, Common Data Elements (CDEs) are standardized questions and response formats used to ensure data comparability across studies. Traditionally, these are defined "top-down" by panels of experts—a process that is slow, expensive, and prone to missing emergent concepts found in actual clinical practice, especially for rare diseases like sarcomas.
Sarcomas are highly heterogeneous, and much of our knowledge about them is buried in thousands of unstructured case reports. This paper investigates whether LLM agents can perform "agentic induction"—reading the literature to propose a schema (bottom-up) and then populating that schema at scale.
Methods
The researchers built a multi-stage workflow using the gpt-oss-120b model and the AG2 agent framework. The process, illustrated in Figure 1, consists of two main phases:
- Corpus-driven Schema Induction:
- Proposal: An agent reads each abstract and proposes potential CDEs (Question, Type, and Value).
- Consolidation: The system uses embedding-based similarity searches and agentic reasoning to merge thousands of raw proposals into a set of "master" CDEs.
- Standardization: A dedicated agent cleans up categorical value sets (e.g., normalizing "Male" and "M").
- Fixed-schema Extraction:
- The final schema (retaining CDEs supported by at least 50 abstracts) is applied to the entire corpus.
- Crucially, the model must provide an evidence span (a direct quote from the text) for every extracted value to ensure traceability and reduce hallucinations.
Datasets
- PubMed: The authors retrieved 58,265 records using sarcoma-related MeSH terms.
- Sarcoma Case Report Corpus: After filtering for single-patient human case reports with abstracts, they settled on a final dataset of 24,577 unique abstracts.
- Evaluation Set: A random sample of 100 abstracts was used for rigorous human expert validation of the extraction accuracy.
Key Insights and Ideas
- Agentic vs. Direct Inference: The authors found that agents are best suited for the "creative" task of schema design (planning and tool use), while direct, constrained LLM calls are more efficient for high-volume data extraction.
- The Frequency vs. Importance Trade-off: The pipeline initially proposed over 72,000 CDEs, but only 53 met the threshold of being mentioned in 50+ abstracts. This highlights a challenge in rare disease research: frequent data points are easy to automate, but "insular criteria" (rare but critical clinical features) might be filtered out by frequency-based thresholds.
- Traceability as a Safeguard: By requiring the model to link every answer to a specific text span, the authors created a built-in verification mechanism. Errors were often caught when the model couldn't find a matching span for its proposed answer.
- Schema Quality vs. Extraction Accuracy: Interestingly, human experts were much more critical of the schema (the questions themselves) than the extraction (the answers). This suggests that LLMs are currently better at following a schema than designing a perfect one from scratch.
Key Results
The workflow's progression from raw proposals to a refined schema is detailed in Figure 2.
- Extraction Performance: In the 100-abstract validation set, the LLM achieved 94.9% exact agreement with human experts for non-empty values.
- Schema Induction: The agent successfully consolidated 72,909 raw proposals into 1,570 master CDEs.
- Expert Review: Two medical professionals approved 83.0% of the retained CDEs for relevance, but only 56.6% for correctness, citing issues like overlapping fields (e.g., multiple CDEs for patient age).
- Data Coverage: As shown in Figure 3, coverage varied significantly. "Patient Sex" was extracted in 80.6% of cases, while more specific data like "Tumor Dimensions" appeared in only 10.5%.
Limitations and Future Work
- Abstract Limitations: Abstracts often omit granular details like specific chemotherapy dosages or long-term follow-up data, which are only available in full-text reports.
- Semantic Overlap: Despite two consolidation passes, the agent still produced redundant CDEs. Future iterations need better logic for identifying when two differently worded questions ask for the same clinical variable.
- Complex Relations: The model struggled with anatomical locations and temporal concepts (e.g., distinguishing between the primary tumor location and a metastatic site).
- Human-in-the-loop: The authors conclude that the most viable path forward is an agent-assisted approach: let the AI draft the schema and perform the bulk extraction, but keep experts in the loop to curate the schema before it is deployed at scale.
Summary updated