Desperately seeking data: knowledge base-database links.

George Hripcsak, Stephen B. Johnson, Paul D. ClaytonPubMed 1993

TL;DR

This paper quantifies the significant overhead associated with linking knowledge-based systems (KBSs) to clinical databases. It reveals that the "glue code" for these links consumes more resources in terms of development, maintenance, and execution time than the actual medical knowledge within the system.

What Problem Does This Paper Address?

The widespread adoption of knowledge-based systems (KBSs) in clinical settings has been hampered by a critical challenge: effectively linking these systems to existing clinical databases. While KBSs promise to enhance patient care through alerts, interpretations, and research screening, their practical utility is often limited by the difficulty of accessing and integrating real-time patient data. This paper addresses the substantial, often underestimated, effort required to establish and maintain these KBS-database connections, which is a major bottleneck to realizing the full potential of clinical informatics.

Methods

The authors analyzed the Medical Logic Modules (MLMs) within the Columbia-Presbyterian Medical Center's (CPMC) clinical event monitor. They focused on four key aspects of the MLMs:

  • Code Size: They quantified the number of statements and "tokens" (defined as numbers, times, terms, strings, identifiers, and operators) within different MLM slots, specifically comparing the data slot (which defines KBS-database links) to the logic slot (which contains the medical knowledge).
  • Modifications: They tracked the number of code modifications made to each MLM slot over time using the Unix diff utility to measure changes between versions.
  • Execution Time: They measured the elapsed time for different components of MLM execution, breaking it down into database queries, database updates, and MLM logic processing.

Datasets

Not discussed in the paper. The study uses the existing knowledge base and clinical database at Columbia-Presbyterian Medical Center (CPMC).

Key Insights and Ideas

  • The "Glue" is the Bottleneck: The most surprising finding is that the code dedicated to linking the KBS to the database (the data slot) is not only substantial in size but also requires significantly more maintenance and consumes the vast majority of execution time compared to the actual medical logic.
  • Maintenance Nightmare: The data slot undergoes roughly twice as many modifications per version as the logic slot, indicating a much higher rate of change and a greater maintenance burden.
  • Performance Drain: Database queries and updates, which are part of the KBS-database link, account for an overwhelming 97.2% of the total elapsed execution time for the MLMs. This means the system spends almost all its time fetching and processing data, leaving very little for actual reasoning.
  • Complexity is Inevitable: Even with standardized syntax like Arden, institutions have unique database structures and terminologies, necessitating custom query development for each link, making it a critical and difficult component regardless of the KBS type.

Key Results

  • Code Size: The data slot had more tokens (122 vs. 76) than the logic slot, despite having a similar number of statements (5.8 vs. 8.8).
  • Modifications: The data slot had significantly more modifications (1.97 per version) than the logic slot (1.06 per version).
  • Execution Time: Database queries and updates accounted for 97.2% of the total elapsed execution time, while MLM logic represented only 2.8%. Specifically, query execution averaged 194.9 milliseconds, while logic averaged a mere 8.4 milliseconds.

Limitations and Future Work

  • Vague Definitions: The authors acknowledge that the distinction between what constitutes a KBS-database link and MLM logic can be vague, potentially leading to a conservative estimate of the link's impact. For instance, data filtering that could be argued as part of the link was included in the logic slot.
  • Future Work: The paper suggests that while no simple solution exists to reduce the effort of generating and maintaining these links, providing a supportive environment with tools for vocabulary selection, query reuse, interactive query testing, and MLM debugging is crucial. They also highlight the need for systems to adapt as patient data, coding schemes, and systems change, suggesting automated monitoring for significant changes in MLM activity.

Summary updated

Abstract

Linking a knowledge-based system (KBS) to a clinical database is a difficult task, but critical if such systems are to achieve widespread use. The Columbia-Presbyterian Medical Center's clinical event monitor provides alerts, interpretations, research screening, and quality assurance functions for the center. Its knowledge base consists of Arden Syntax Medical Logic Modules (MLMs). The knowledge base was analyzed in order to quantify the use and impact of KBS-database links. The MLM data slot, which contains the definition of these links, had almost as many statements (5.8 vs. 8.8, ns with p = 0.15) and more tokens (122 vs. 76, p = 0.037) than the logic slot, which contains the actual medical knowledge. The data slot underwent about twice as many modifications over time as the logic slot (3.0 vs. 1.6 modifications/version, p = 0.010). Database queries and updates accounted for 97.2% of the MLM's total elapsed execution time. Thus, KBS-database links consume substantial resources in an MLM knowledge base, in terms of coding, maintenance, and performance.
BibTeX
@inproceedings{hripcsak1993desperately,
  author = {George Hripcsak and Stephen B. Johnson and Paul D. Clayton},
  title = {{Desperately seeking data: knowledge base-database links.}},
  booktitle = {PubMed},
  year = {1993},
}
Figure