Abstract
Machine learning models for clinical prediction rely on structured data extracted from Electronic Medical Records (EMRs), yet this process remains dominated by hardcoded, database-specific pipelines for cohort definition, feature selection, and code mapping. These manual efforts limit scalability, reproducibility, and cross-institutional generalization. To address this, we introduce EMR-AGENT (Automated Generalized Extraction and Navigation Tool), an agent-based framework that replaces manual rule writing with dynamic, language model-driven interaction to extract and standardize structured clinical data. Our framework automates cohort selection, feature extraction, and code mapping through interactive querying of databases. Our modular agents iteratively observe query results and reason over schema and documentation, using SQL not just for data retrieval but also as a tool for database observation and decision making. This eliminates the need for hand-crafted, schema-specific logic. To enable rigorous evaluation, we develop a benchmarking codebase for three EMR databases (MIMIC-III, eICU, SICdb), including both seen and unseen schema settings. Our results demonstrate strong performance and generalization across these databases, highlighting the feasibility of automating a process previously thought to require expert-driven design.
Introduction: The Labyrinth of Medical Databases
Building machine learning models for clinical prediction—such as predicting sepsis or mortality—is currently bottlenecked by a massive data preprocessing challenge. Every hospital system stores its Electronic Medical Records (EMRs) differently. For instance, "heart rate" might be recorded as itemid=211 in one database and as a column named heartrate in another.
Currently, researchers must manually "reverse-engineer" these database schemas, writing thousands of lines of custom SQL code for every new project. This manual approach is non-scalable, error-prone, and leads to significant reproducibility issues. EMR-AGENT aims to replace these rigid, manual pipelines with a dynamic, agentic system capable of exploring databases much like a human data scientist.
Methods: The EMR-AGENT Framework
EMR-AGENT utilizes two specialized Large Language Model (LLM) agents to navigate the complexities of clinical data:
- Cohort and Feature Selection Agent (CFSA): This agent identifies specific patient populations (e.g., "males aged 70-80 with a first-time ICU admission") and extracts relevant demographic and clinical variables.
- Code Mapping Agent (CMA): This agent standardizes clinical measurements by searching the database to find specific codes or labels that correspond to concepts like "glucose" or "systolic blood pressure."
The Observe-Reason-Act Loop
The core innovation of EMR-AGENT is its iterative reasoning process. Unlike standard "Text-to-SQL" systems that attempt to generate a perfect query in a single pass, EMR-AGENT employs an Observe-Reason-Act loop:
- Schema Linking: The agent reads database metadata and expert "memos" to understand table structures.
- SQL-based Observation: It executes exploratory queries (e.g.,
SELECT DISTINCT) to verify the contents of a column before finalizing an extraction plan. - Error Feedback: If a query fails or returns unexpected results, the agent analyzes the error message, adjusts its logic, and retries.
Experimental Evaluation
The system was evaluated using a new benchmarking codebase, PreCISE-EMR, across three major ICU databases:
- MIMIC-III: A single-center US database.
- eICU: A multi-center US database with a distinct structure.
- SICdb: A European database released after the training cutoff of the LLMs used, serving as a rigorous test for generalization to unseen environments.
Results
EMR-AGENT significantly outperformed traditional Text-to-SQL baselines across all metrics, demonstrating its ability to handle complex, real-world medical data structures.
Key findings include:
- Cohort Selection: Achieved F1 scores of 0.94 on MIMIC-III and 0.93 on eICU. Notably, it maintained a strong 0.81 on the unseen SICdb.
- Code Mapping: The agent improved mapping F1 scores by 0.30 to 0.52 over the best existing competitors, which often failed entirely on these tasks.
- The Importance of Exploration: Ablation studies confirmed that "SQL-based Observation" is critical; without the ability to "look before it leaps," performance drops significantly.
Conclusion
EMR-AGENT demonstrates that LLMs have reached a level of reasoning where they can apply general clinical knowledge to unfamiliar data structures. By treating SQL as a tool for exploration rather than just a final output, the framework automates a process previously thought to require expert-driven design. While human validation remains necessary for high-stakes applications, EMR-AGENT represents a significant step toward scalable and reproducible clinical machine learning.
Summary updated