Automatic Coding at Scale: Design and Deployment of a Nationwide System for Normalizing Referrals in the Chilean Public Healthcare System

Fabián Villena, Matías Rojas, Felipe Arias, Jorge Pacheco, Paulina Vera, Jocelyn DunstanACL 2023

TL;DR

This paper presents a novel two-step system for automatically coding diseases in Chilean public healthcare referrals, combining a state-of-the-art Named Entity Recognition (NER) model with an Elasticsearch-based search engine. The system achieves impressive performance, nearing that of human experts, and has been deployed nationwide to improve the management of healthcare resources.

What Problem Does This Paper Address?

The Chilean public healthcare system generates a massive volume of patient referrals, many of which are in unstructured text format. Manually assigning standardized disease codes (like ICD-10) to these referrals is a time-consuming, error-prone, and resource-intensive process. This manual burden diverts clinical personnel from patient care and hinders efficient data analysis for epidemiological studies and resource management. Existing automated systems often struggle with the nuances of clinical language, lack context-specific rules, and may not be readily adaptable to different languages or healthcare systems. This paper aims to bridge this gap by developing a scalable, automated system for disease coding within the Chilean context.

Methods

The proposed system employs a two-stage approach:

  1. Named Entity Recognition (NER): The first stage utilizes a deep neural network model called Multiple LSTM-CRF (MLC). This model leverages domain-specific contextualized embeddings (both character-level and static word embeddings trained on clinical data) to accurately identify disease mentions within the unstructured referral text. The LSTM layers capture long-distance dependencies, and the CRF layer assigns the most probable labels to words, marking the beginning and end of disease entities.
  2. Search Engine for Entity Linking: Once disease mentions are identified, the second stage uses an Elasticsearch-based search engine to link these mentions to the most appropriate ICD-10 codes. This engine indexes a comprehensive set of ICD-10 terminology, including canonical descriptions, synonyms, and abbreviations, allowing for a robust matching process that simulates how human coders might use various resources.

Datasets

  • Chilean Waiting List Corpus: The system was trained and evaluated on a dataset of 10,000 referrals from the Chilean Waiting List. This corpus was previously consolidated by clinical experts and represents a valuable resource for medical entity recognition in Spanish.
  • ICD-10 Terminology: Various components of the ICD-10 terminology were used, including the tabular list, alphabetical index, UMLS (for Spanish definitions), DEIS abbreviations, and the IRIS dictionary.

Key Insights and Ideas

  • Hybrid Approach is Key: Combining deep learning for robust entity recognition with a knowledge-rich search engine for entity linking proves more effective than purely end-to-end deep learning models, as it allows for the incorporation of domain-specific knowledge.
  • Leveraging Existing Terminology Resources: The system's success hinges on its ability to effectively utilize a wide array of ICD-10 related resources, mimicking the tools used by human coders, which helps overcome the discrepancy between free text and standardized codes.
  • Context Matters: The MLC model's ability to capture long-distance dependencies in clinical text is crucial for accurately identifying disease mentions, especially in complex or abbreviated descriptions.
  • Scalability and Deployment: The paper emphasizes not just the model's performance but also its practical deployment within the Chilean Ministry of Health, highlighting the feasibility of large-scale implementation.

Key Results

  • Performance: The system achieved a Mean Average Precision (MAP) score of 0.63 at the subcategory level and 0.83 at the category level.
  • Comparison to Experts: These results are comparable to the agreement among human clinical experts, who achieved a MAP of 0.75 for subcategory and 0.83 for category level. This suggests the automated system can perform at a level close to human experts, despite the inherent subjectivity and challenges in manual coding.
  • Scale: The system was used to code a total of 18,716,629 referrals from the Chilean Waiting List, demonstrating its efficiency and effectiveness at scale.
  • Deployment: The system has been deployed in production within the Department of Health Statistics and Information Systems of the Ministry of Health of Chile.

Limitations and Future Work

  • Entity-Level vs. Referral-Level Coding: The current manual coding experiment for evaluation was performed at the referral level, not the entity level. This means the performance of the NER model itself wasn't directly evaluated in this specific experiment. Future work could use specialized software (like INCEPTION) to enable entity-level evaluation.
  • Scope of Entities: Currently, the system focuses on coding diseases. Future research could extend its capabilities to include other entity types, such as clinical procedures or clinical findings.
  • Specialty-Specific Performance: While overall performance is strong, the paper notes variations in performance across different medical specialties, with some specialties showing significantly lower MAP scores. Further analysis and potential fine-tuning for these specific areas could improve overall accuracy.
  • Data Privacy and On-Premise Deployment: Due to strict data privacy regulations, the system was deployed on-premise. While this ensures data security, it might limit the use of cloud-based computational resources for even faster processing.

Summary updated

BibTeX
@inproceedings{villena2023automatic,
  author = {Fabián Villena and Matías Rojas and Felipe Arias and Jorge Pacheco and Paulina Vera and Jocelyn Dunstan},
  title = {{Automatic Coding at Scale: Design and Deployment of a Nationwide System for Normalizing Referrals in the Chilean Public Healthcare System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure