Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysis

Rafi Al Attrach, Pedro Miguel Moreira, Rajna Fani, Renato Umeton, Amelia Fiske, Jack GallifantarXiv 2025 · DOI · arXiv

Abstract

Large-scale clinical databases offer opportunities for medical research, but their complexity creates barriers to effective use. The Medical Information Mart for Intensive Care (MIMIC-IV), one of the world's largest open-source electronic health record databases, traditionally requires both SQL proficiency and clinical domain expertise. We introduce M3, a system that enables natural language querying of MIMIC-IV data through the Model Context Protocol. With a single command, M3 retrieves MIMIC-IV from PhysioNet, launches a local SQLite instance or connects to hosted BigQuery, and allows researchers to pose clinical questions in plain English. We evaluated M3 using one hundred questions from the EHRSQL 2024 benchmark with two language models: the proprietary Claude Sonnet 4 achieved 94% accuracy, while the open-source gpt-oss-20B (deployable locally on consumer hardware) achieved 93% accuracy. Both models translate natural language into SQL, execute queries against MIMIC-IV, and return structured results alongside the underlying query for verification. Error analysis revealed that most failures stemmed from complex temporal reasoning or ambiguous question phrasing rather than fundamental architectural limitations. The comparable performance of a smaller open-source model demonstrates that privacy-preserving local deployment is viable for sensitive clinical data analysis. M3 lowers technical barriers to critical care data analysis while maintaining security through OAuth2 authentication, query validation, and comprehensive audit logging.

Introduction

Large-scale Electronic Health Record (EHR) databases like MIMIC-IV (Medical Information Mart for Intensive Care) are gold mines for medical research, containing de-identified data from hundreds of thousands of ICU stays. However, they are notoriously difficult to use. To extract meaningful insights, a researcher typically needs both deep clinical domain expertise and high proficiency in SQL. This "interdisciplinary gap" often prevents clinical experts from directly querying data, while data scientists may lack the clinical nuance to interpret complex relational schemas correctly. M3 addresses this by providing a natural language interface that bridges these two worlds.

System Architecture: The M3 Framework

The M3 system is a Python-based server application that acts as a secure intermediary between Large Language Models (LLMs) and clinical databases. The core innovation is the use of the Model Context Protocol (MCP), which provides a standardized way for AI models to interact with external tools and data sources.

The M3 Server architecture utilizes a layered approach with MCP to bridge the gap between clients and clinical databases.

M3 implements a layered architecture to ensure both flexibility and security:

  1. Data Access Layer: Supports both local SQLite (for rapid prototyping) and Google BigQuery (for full-scale research).
  2. Security Middleware: Implements OAuth2 authentication and a defensive SQL validation system that blocks non-read-only queries.
  3. Two-Tiered Tooling: Includes core tools for schema discovery and domain-specific clinical tools that abstract away complex joins for common tasks like retrieving lab results or demographic distributions.

Evaluation and Results

The system was evaluated using 100 answerable questions from the EHRSQL 2024 Benchmark against the MIMIC-IV (v3.1) dataset. The results demonstrate that conversational interfaces are highly effective for clinical data retrieval:

  • Claude Sonnet 4: Achieved 94% accuracy.
  • gpt-oss-20B: Achieved 93% accuracy.
Example of a complex clinical query result: Systolic Blood Pressure trends in ICU patients on vasopressors, generated via natural language query.

The negligible performance gap between the proprietary model and the open-source model is a major win for medical informatics. It proves that high-accuracy clinical data analysis can be performed locally on consumer-grade hardware, keeping sensitive data entirely within a secure local environment.

Discussion and Safety

Error analysis revealed that failures were rarely due to incorrect SQL generation but rather stemmed from semantic ambiguity in natural language. For example, a request for "days since last stay" requires the model to decide if that refers to the start or end of the hospitalization. To maintain analytical rigor, M3 returns the generated SQL alongside the data results, allowing researchers to verify the AI's logic.

Conclusion and Future Work

M3 moves clinical research closer to a world where a doctor can "chat" with their data, turning months of SQL-heavy data engineering into a few minutes of natural conversation. Future iterations will focus on expanding to other datasets like eICU, integrating Retrieval-Augmented Generation (RAG) for biomedical literature grounding, and implementing conversational refinement loops to resolve linguistic ambiguities.

Summary updated

Abstract

Large-scale clinical databases offer opportunities for medical research, but their complexity creates barriers to effective use. The Medical Information Mart for Intensive Care (MIMIC-IV), one of the world's largest open-source electronic health record databases, traditionally requires both SQL proficiency and clinical domain expertise. We introduce M3, a system that enables natural language querying of MIMIC-IV data through the Model Context Protocol. With a single command, M3 retrieves MIMIC-IV from PhysioNet, launches a local SQLite instance or connects to hosted BigQuery, and allows researchers to pose clinical questions in plain English. We evaluated M3 using one hundred questions from the EHRSQL 2024 benchmark with two language models: the proprietary Claude Sonnet 4 achieved 94% accuracy, while the open-source gpt-oss-20B (deployable locally on consumer hardware) achieved 93% accuracy. Both models translate natural language into SQL, execute queries against MIMIC-IV, and return structured results alongside the underlying query for verification. Error analysis revealed that most failures stemmed from complex temporal reasoning or ambiguous question phrasing rather than fundamental architectural limitations. The comparable performance of a smaller open-source model demonstrates that privacy-preserving local deployment is viable for sensitive clinical data analysis. M3 lowers technical barriers to critical care data analysis while maintaining security through OAuth2 authentication, query validation, and comprehensive audit logging.
BibTeX
@misc{attrach2025conversational,
  author = {Rafi Al Attrach and Pedro Miguel Moreira and Rajna Fani and Renato Umeton and Amelia Fiske and Jack Gallifant},
  title = {{Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysis}},
  year = {2025},
  eprint = {2507.01053},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.01053},
  howpublished = {arXiv},
}
Figure