TL;DR
This paper investigates the quality of functional status documentation in Electronic Health Records (EHRs) across different healthcare institutions. It reveals significant variations in how this crucial patient information is recorded, largely due to the use of different "instrument-assisted documentation" (iDoc) methods, impacting the reliability of data for research and AI applications. The findings underscore the critical need for robust data quality assessments before utilizing EHR data for downstream tasks.
What Problem Does This Paper Address?
The secondary use of Electronic Health Records (EHRs) for research, decision support, and AI development is a rapidly growing field. However, a major hurdle is the variable quality of data within EHRs, particularly in unstructured text. This paper focuses on "functional status" documentation – information about a patient's ability to perform daily activities (like dressing, bathing, or managing medications). This data is vital for understanding patient health, especially in aging populations. The problem is that this documentation isn't standardized; it's influenced by the EHR system itself, the clinical workflow, and the specific tools used by healthcare providers. These variations, termed "instrument-assisted documentation" (iDoc), can lead to inconsistent and unreliable data, posing a significant challenge for Natural Language Processing (NLP) applications that aim to extract and analyze this information.
Methods
The researchers employed a convergent parallel design, combining both quantitative and qualitative methods to get a comprehensive understanding of the problem.
- Qualitative: They conducted interviews with clinicians to understand the narrative documentation process and how functional status information is captured in different clinical settings.
- Quantitative:
- iDoc Identification: They systematically identified functional status-related expressions in clinical notes. This involved corpus annotation, measuring textual similarity between expressions, and manually reviewing those with high similarity. They then developed symbolic methods to automatically identify these iDoc expressions.
- Information Quality Assessment: They adopted the AIMQ (Assessment of Information Quality) framework to evaluate intrinsic information quality (accuracy) and contextual information. Intrinsic quality was assessed through agreement between similar iDoc expressions and by correlating iDoc data with diagnoses of mild cognitive impairment (MCI) and dementia. Contextual quality involved examining the distribution of iDoc expressions across different document sections, event types, and hospital services.
- NLP Implications: They analyzed how different iDoc representations (from EHR smart forms, questionnaires, and PT/OT templates) impact NLP pipelines, evaluating their naturality of language and their effect on NLP components like section detectors and context algorithms.
Datasets
The study utilized data from the Mayo Clinic Study of Aging (MCSA) cohort. This is a population-based cohort study focused on cognitive aging, initiated in 2004, comprising over 6,000 participants. For this specific study, a random sample of 673 patients was selected, with visit times ranging from December 2004 to February 2020. The data was collected from three different healthcare institutions.
Key Insights and Ideas
- iDoc is Ubiquitous and Heterogeneous: The study found that "instrument-assisted documentation" (iDoc) is a prevalent way functional status is recorded in EHRs, but its form and content vary dramatically.
- Source Matters: The type of instrument used (e.g., EHR smart forms, questionnaires, physical/occupational therapy templates) significantly influences the language, information quality, and how easily NLP can process the data.
- Naturalness Varies: EHR smart forms tend to have more natural language, while PT/OT templates often result in more semi-structured text, posing challenges for NLP.
- Context is King (and Inconsistent): The context in which iDoc is documented (e.g., hospital service, event type) differs greatly across institutions and iDoc types, highlighting that iDoc is highly context-specific.
- Semantic Conflicts Exist: Even within the same document, conflicting information about a patient's functional status can be present (e.g., reporting both difficulty with dressing and being independent). This is a critical issue for data accuracy.
- NLP Portability is Challenged: The significant variations in iDoc documentation across institutions and instruments create a major hurdle for developing NLP models that can be easily deployed and perform reliably in different healthcare settings.
Key Results
- Four Prevalent iDoc Expressions: The study identified four main types of iDoc expressions, stemming from three distinct instrument types (Epic smart form, questionnaire, PT/OT templates).
- Quality Varies:
- iDoc (i) and (ii) showed moderate to high validity for predicting MCI and dementia diagnoses (PPV of 0.735 and 0.874 for iDoc (i), NPV of 0.883 and 0.980 for iDoc (ii)).
- Agreement between semantically similar iDoc (iii) expressions was high (0.963 fl-score).
- iDoc (iv) quality was not assessed due to its limited direct semantic contribution.
- Contextual Heterogeneity: The number of unique hospital services associated with iDoc expressions ranged from 3 to 183, demonstrating significant contextual differences.
- NLP Pipeline Impact: Questionnaire forms and PT/OT templates require more complex preprocessing steps for NLP compared to EHR smart forms, which have minimal impact on existing pipelines. Traditional lexicon-based NLP approaches failed to capture all iDoc types.
Limitations and Future Work
- Limited Qualitative Data: Interviews were primarily conducted at one institution due to recruitment challenges, potentially limiting the representativeness of the qualitative findings.
- Generalizability: The study was conducted on three sites with a single case scenario, which limits the generalizability of the findings.
- Future Work: The authors suggest collaborating with national data consortiums to establish standardized operating procedures and best practices for quality assessment. They also aim to broaden the study scope by involving more institutions and case studies. They recommend applying a suite of information quality assessments (accessibility, intrinsic, representational, and contextual IQ) prior to using EHR data for secondary purposes. They also propose exploring statistics-based sub-language analysis to improve NLP algorithm portability.
Summary updated