TL;DR
This paper quantifies the significant overhead associated with linking knowledge-based systems (KBSs) to clinical databases. It reveals that the "glue code" for these links consumes more resources in terms of development, maintenance, and execution time than the actual medical knowledge within the system.
What Problem Does This Paper Address?
The widespread adoption of knowledge-based systems (KBSs) in clinical settings has been hampered by a critical challenge: effectively linking these systems to existing clinical databases. While KBSs promise to enhance patient care through alerts, interpretations, and research screening, their practical utility is often limited by the difficulty of accessing and integrating real-time patient data. This paper addresses the substantial, often underestimated, effort required to establish and maintain these KBS-database connections, which is a major bottleneck to realizing the full potential of clinical informatics.
Methods
The authors analyzed the Medical Logic Modules (MLMs) within the Columbia-Presbyterian Medical Center's (CPMC) clinical event monitor. They focused on four key aspects of the MLMs:
- Code Size: They quantified the number of statements and "tokens" (defined as numbers, times, terms, strings, identifiers, and operators) within different MLM slots, specifically comparing the
dataslot (which defines KBS-database links) to thelogicslot (which contains the medical knowledge). - Modifications: They tracked the number of code modifications made to each MLM slot over time using the Unix
diffutility to measure changes between versions. - Execution Time: They measured the elapsed time for different components of MLM execution, breaking it down into database queries, database updates, and MLM logic processing.
Datasets
Not discussed in the paper. The study uses the existing knowledge base and clinical database at Columbia-Presbyterian Medical Center (CPMC).
Key Insights and Ideas
- The "Glue" is the Bottleneck: The most surprising finding is that the code dedicated to linking the KBS to the database (the
dataslot) is not only substantial in size but also requires significantly more maintenance and consumes the vast majority of execution time compared to the actual medical logic. - Maintenance Nightmare: The
dataslot undergoes roughly twice as many modifications per version as thelogicslot, indicating a much higher rate of change and a greater maintenance burden. - Performance Drain: Database queries and updates, which are part of the KBS-database link, account for an overwhelming 97.2% of the total elapsed execution time for the MLMs. This means the system spends almost all its time fetching and processing data, leaving very little for actual reasoning.
- Complexity is Inevitable: Even with standardized syntax like Arden, institutions have unique database structures and terminologies, necessitating custom query development for each link, making it a critical and difficult component regardless of the KBS type.
Key Results
- Code Size: The
dataslot had more tokens (122 vs. 76) than thelogicslot, despite having a similar number of statements (5.8 vs. 8.8). - Modifications: The
dataslot had significantly more modifications (1.97 per version) than thelogicslot (1.06 per version). - Execution Time: Database queries and updates accounted for 97.2% of the total elapsed execution time, while MLM logic represented only 2.8%. Specifically, query execution averaged 194.9 milliseconds, while logic averaged a mere 8.4 milliseconds.
Limitations and Future Work
- Vague Definitions: The authors acknowledge that the distinction between what constitutes a KBS-database link and MLM logic can be vague, potentially leading to a conservative estimate of the link's impact. For instance, data filtering that could be argued as part of the link was included in the logic slot.
- Future Work: The paper suggests that while no simple solution exists to reduce the effort of generating and maintaining these links, providing a supportive environment with tools for vocabulary selection, query reuse, interactive query testing, and MLM debugging is crucial. They also highlight the need for systems to adapt as patient data, coding schemes, and systems change, suggesting automated monitoring for significant changes in MLM activity.
Summary updated