TL;DR
This paper introduces emrKBQA, a large-scale dataset for answering physician questions from structured electronic health records (EHRs). It addresses the challenge of building robust clinical question-answering systems by providing a benchmark for mapping natural language questions to logical forms and retrieving answers from structured data.
What Problem Does This Paper Address?
The increasing adoption of Electronic Health Records (EHRs) presents a massive opportunity for improving clinical decision-making. Physicians often need to query patient EHRs for specific information, but current systems struggle with complex, natural language questions. Developing machine learning models for this task is hampered by the lack of large, annotated datasets. Existing datasets like emrQA focus on unstructured clinical notes, but structured EHR data (like lab results, prescriptions) is often more reliable for factual information. This paper aims to bridge this gap by creating a dataset that facilitates question answering over the structured portion of EHRs.
Methods
The core of this work is the creation of the emrKBQA dataset. The process is semi-automated, building upon the methodology of the earlier emrQA dataset. Key steps include:
- Question Templating and Slot-Filling: Physician questions were collected and transformed into templates with entity placeholders. These templates were then "slot-filled" with actual entities from a knowledge base (KB) derived from the MIMIC-III database.
- Expert-Driven Refinement: Unlike emrQA, emrKBQA incorporates expert medical knowledge to filter slot-filling. This ensures that the generated questions are more realistic and relevant to clinical practice (e.g., filtering medications by drug type).
- Logical Form Generation: For each question, a corresponding logical form is generated. These logical forms are a structured, query-language-agnostic representation of the information needed to answer the question. They are derived from a schema that maps to the entities and attributes found in the EHR.
- Benchmarking Semantic Parsing: The authors propose a two-step approach for question answering: first, semantic parsing to convert natural language questions into logical forms, and second, mapping these logical forms to SQL queries to retrieve answers from the KB. They benchmark semantic parsing models using both naive and more challenging paraphrase-level splitting schemes.
Datasets
- emrKBQA: This is the primary contribution of the paper. It's a newly introduced, large-scale dataset consisting of over 940,000 question, logical form, and answer triplets. It includes 389 question types (templates) and approximately 7.5 paraphrases per question type. The dataset is derived from the structured portion of the MIMIC-III database.
- emrQA: The paper builds upon the methodology of the emrQA dataset, which focused on unstructured clinical notes. emrKBQA extends this by focusing on structured EHR data.
- MIMIC-III KB: The structured data for generating emrKBQA is sourced from the MIMIC-III database, a widely used critical care database.
Key Insights and Ideas
- Bridging Structured and Unstructured Data: The paper highlights the complementary nature of structured and unstructured EHR data for comprehensive question answering. While structured data excels at factual recall (lab values, prescriptions), unstructured data is better for capturing complex relations, symptoms, and temporal events. emrKBQA focuses on the structured side, paving the way for future systems that can query both.
- The Importance of Expert Input: The refinement of the slot-filling process using medical expert input is crucial. This step significantly improves the realism and clinical relevance of the generated questions, addressing a limitation of purely automated generation.
- Challenging Semantic Parsing: The introduction of paraphrase-level splitting schemes for evaluation reveals the difficulty of training models that can generalize to unseen paraphrases. This is a more realistic scenario for real-world applications.
- Logical Forms as a Unifying Factor: The use of logical forms, derived from a common schema, allows for a consistent approach to question answering across both structured and unstructured data sources in the future.
Key Results
- Dataset Size: emrKBQA contains 940,713 question-answer pairs derived from 100 patients, 389 question templates, and 52 question/logical form templates.
- Question Distribution: The majority of questions in emrKBQA are related to "Test" results (90.9%), followed by "Medication" (7.8%) and "Condition" (1.2%). This reflects the nature of structured data and the source of the questions.
- Semantic Parsing Performance:
- In a naive splitting scheme, a baseline seq2seq model with a copy mechanism achieved high Exact Match (EM) accuracy of 0.95 and Denotation Accuracy of 0.96.
- With paraphrase-level splits, performance dropped, with the baseline achieving 0.83 EM and 0.84 Denotation Accuracy.
- A more advanced model (ParaGen+ParaDetect) showed similar performance to the baseline on paraphrase splits (0.82 EM, 0.82 Denotation Accuracy), suggesting potential for further hyperparameter tuning.
- Error Analysis: The most common errors in semantic parsing were attributed to:
- Questions with multiple entities (47%).
- Confusion between entity types (28%).
- Incorrect entity span recognition (12%).
Limitations and Future Work
- Dataset Limitations:
- The question set is limited by the initial pool of questions collected from polled physicians.
- The semi-automated generation process can lead to some slot-filled questions that might not be asked in a real clinical setting.
- There's redundancy in "question form" due to slot filling, with many instances of the same template using different entities.
- Relationships between conditions and symptoms, or conditions and treatments, are not well-captured in structured data and thus not fully represented in emrKBQA.
- Future Work:
- Increase the range of question types and granularity.
- Incorporate domain knowledge for better slot-filling (e.g., using word/phrase synonyms).
- Improve temporal question classification.
- Generate more question paraphrases using automated methods.
- Expand the dataset to include more patients and explore patient cohorts with specific criteria.
- Develop models that can answer questions by querying both structured and unstructured EHR data.
Summary updated