A ModelOps-based Framework for Intelligent Medical Knowledge Extraction

Hongxin Ding, Peinie Zou, Zhiyuan Wang, Junfeng Zhao, Yasha Wang, Qiang Zhou

Introduction

Extracting structured information—such as diseases, symptoms, and drug relations—from unstructured medical text is a cornerstone of modern clinical decision support. However, the path from raw clinical notes to actionable insights is fraught with technical barriers. Researchers from Peking University have identified two primary hurdles: the "Integration Tax" and the "Expertise Gap."

The Integration Tax refers to the $N \times M$ complexity problem, where every new dataset ($N$) paired with a new model ($M$) requires custom "glue code" for data formatting and preprocessing. Simultaneously, the Expertise Gap prevents clinicians and medical researchers—who possess the domain knowledge but often lack deep engineering skills—from effectively managing GPU resources, training loops, and model selection. To address these challenges, this paper proposes a ModelOps-based framework designed to democratize medical knowledge extraction through a low-code, automated architecture.

The Proposed Framework Architecture

The framework is built upon a three-pillar architecture designed to decouple data management from model execution, thereby reducing the coding burden from $N \times M$ to $N + M$.

1. Multi-layer Callback Abstraction

At the heart of the system is a three-layer "plugin" system consisting of Data Retrieval, Data Cleaning, and Feature Extraction. By abstracting these processes, a dataset only needs to be "wrapped" once to become compatible with any model in the library. This modularity allows models to be "hot-swappable," enabling researchers to test different architectures without rewriting data-loading logic.

2. Automated ModelOps Pipeline

The system utilizes a distributed architecture comprising a Service Gateway, an AI Server, and various Workers. This setup automates resource allocation by binding tasks to available GPUs and managing the experiment lifecycle. Using YAML configurations and a reflection mechanism, the framework handles training, evaluation, and version control, providing a "one-click" experience for the user.

3. Similarity-Based Model Recommendation

To bridge the expertise gap, the framework includes an intelligent recommendation engine. When a user introduces a new dataset, the system calculates its "distributional similarity" to existing datasets in the knowledge base. This process treats datasets as probability distributions rather than simple collections of text, capturing the linguistic "texture" and technical density of the data.

Intelligent Model Recommendation Engine

Selecting the optimal model for a specific medical dataset is often a matter of trial and error. The authors propose a more scientific approach using BERT-generated embeddings and four distinct distributional metrics:

  • Maximum Mean Discrepancy (MMD)
  • Fréchet Bert Distance (FBD)
  • Precision-Recall (PR) distance
  • Mauve distance

Recognizing that no single metric is universally perfect, the framework employs a rank-sum ratio to fuse these metrics. This ensemble approach ensures a more robust recommendation by balancing the strengths and weaknesses of individual distance measures.

Experimental Results

The framework was validated using a knowledge base of 12 extraction datasets and 8 models. For the recommendation engine, the authors tested seven diverse Named Entity Recognition (NER) datasets across various domains (e.g., Finance, Ecommerce, and Social Media).

Key findings include:

  • Recommendation Accuracy: The rank-sum ratio method achieved a 0.81 accuracy in identifying the best-performing model for a given dataset, significantly outperforming the best single metric (0.76).
  • Operational Efficiency: The full-stack implementation (built with Vue.js, Django, and gRPC) successfully automated experiment tracking via TensorBoard and Grafana, allowing non-experts to manage complex AI lifecycles through a simple web interface.

Discussion and Future Work

While the framework represents a significant step toward low-code medical AI, certain limitations remain. The similarity experiments were primarily conducted on general-domain datasets; future research must verify if these metrics maintain their predictive power within highly specialized clinical sub-specialties. Additionally, calculating metrics like MMD is computationally intensive, requiring sub-sampling of large datasets which may overlook "long-tail" linguistic features.

Future iterations of the framework aim to expand beyond Named Entity Recognition to include complex Relation Extraction (RE) and joint extraction tasks, further empowering clinicians to extract value from the vast wealth of unstructured medical data.

Summary updated

Abstract

Extracting medical knowledge from healthcare texts enhances downstream tasks like medical knowledge graph construction and clinical decision-making. However, the construction and application of knowledge extraction models lack automation, reusability and unified management, leading to inefficiencies for researchers and high barriers for non-AI experts such as doctors, to utilize knowledge extraction. To address these issues, we propose a ModelOps-based intelligent medical knowledge extraction framework that offers a low-code system for model selection, training, evaluation and optimization. Specifically, the framework includes a dataset abstraction mechanism based on multi-layer callback functions, a reusable model training, monitoring and management mechanism. We also propose a model recommendation method based on dataset similarity, which helps users quickly find potentially suitable models for a given dataset. Our framework provides convenience for researchers to develop models and simplifies model access for non-AI experts such as doctors.
BibTeX
@misc{ding2023modelops,
  author = {Hongxin Ding and Peinie Zou and Zhiyuan Wang and Junfeng Zhao and Yasha Wang and Qiang Zhou},
  title = {{A ModelOps-based Framework for Intelligent Medical Knowledge Extraction}},
  year = {2023},
  eprint = {2310.02593},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.02593},
}
Figure