DeepTag: inferring diagnoses from veterinary clinical notes

Allen Nie, Ashley Zehnder, Rodney López Page, Yuhui Zhang, Arturo López Pineda, Manuel A. Rivas, Carlos D. Bustamante, James Zou

Introduction

In human medicine, standardized coding systems like ICD-10 are essential for billing, clinical research, and public health monitoring. Veterinary medicine, however, lacks a similarly robust and automated infrastructure. Most veterinary records are captured as unstructured "SOAP" notes (Subjective, Objective, Assessment, Plan), creating a significant bottleneck for large-scale data analysis. This lack of structured data hinders the identification of patient cohorts for clinical trials and limits cross-species translational research, where pet diseases can offer insights into human health.

DeepTag is a deep learning framework designed to address this gap by automatically assigning standardized SNOMED-CT diagnostic codes to free-text veterinary clinical notes. By combining a hierarchical training objective with a novel "abstention" mechanism, DeepTag bridges the gap between messy clinical narratives and structured medical data.

Methods

DeepTag treats diagnostic coding as a multi-label classification problem using a Bidirectional Long Short-Term Memory (BLSTM) network. This architecture is particularly effective at capturing context within the long, abbreviation-heavy sequences typical of clinical prose.

Overview of the DeepTag pipeline, including hierarchical training and the abstention mechanism.

A key innovation of DeepTag is its Hierarchical Objective. Rather than treating disease codes as independent categories, the model leverages the semantic relationships within the SNOMED-CT hierarchy. By grouping codes into "meta-diseases" and using an $L_2$-based distance objective, the model is forced to place related disease embeddings closer together in vector space. This use of medical hierarchy acts as a form of regularization, improving performance on rarer diseases.

Furthermore, the framework includes a "DeepTag-abstain" layer. This module estimates the model's uncertainty for each note. If the confidence score falls below a certain threshold, the system "abstains" from making a prediction and flags the record for human review, facilitating a high-accuracy hybrid workflow.

Datasets

The model was evaluated using two distinct datasets:

  1. CSU Dataset: 112,558 veterinary notes from Colorado State University, manually annotated with SNOMED-CT codes. This served as the primary training and internal testing set.
  2. PP (Private Practice) Dataset: 586 notes from a private practice in California. This "out-of-domain" set tested the model's ability to generalize across different writing styles and disease distributions.

Results and Discussion

DeepTag demonstrated superior performance over standard baselines, including CNNs and MetaMap. On the CSU test set, it achieved a weighted $F_1$ score of 0.824. While performance dropped on the out-of-domain PP dataset ($F_1$ of 0.634), it remained significantly higher than non-hierarchical models, highlighting the importance of the hierarchical training objective.

Performance gains (F1 and Exact Match scores) as a function of the proportion of cases deferred to human experts.

The abstention mechanism proved highly effective. As shown in the results, deferring the most uncertain cases to human experts leads to a sharp increase in the accuracy of the remaining automated classifications. By abstaining from just 10% of the most difficult cases, the model's performance on the rest of the dataset improved significantly, outperforming simple confidence-based thresholds.

Despite these successes, challenges remain. Vocabulary mismatch—particularly with unseen medical acronyms in private practice—remains a hurdle. Future work will focus on more granular diagnostic codes and optimizing the "cost" of human intervention within the abstention framework. DeepTag provides a scalable blueprint for automating medical coding in domains where structured data is currently scarce.

Summary updated

Abstract

Large scale veterinary clinical records can become a powerful resource for patient care and research. However, clinicians lack the time and resource to annotate patient records with standard medical diagnostic codes and most veterinary visits are captured in free-text notes. The lack of standard coding makes it challenging to use the clinical data to improve patient care. It is also a major impediment to cross-species translational research, which relies on the ability to accurately identify patient cohorts with specific diagnostic criteria in humans and animals. In order to reduce the coding burden for veterinary clinical practice and aid translational research, we have developed a deep learning algorithm, DeepTag, which automatically infers diagnostic codes from veterinary free-text notes. DeepTag is trained on a newly curated dataset of 112,558 veterinary notes manually annotated by experts. DeepTag extends multitask LSTM with an improved hierarchical objective that captures the semantic structures between diseases. To foster human-machine collaboration, DeepTag also learns to abstain in examples when it is uncertain and defers them to human experts, resulting in improved performance. DeepTag accurately infers disease codes from free-text even in challenging cross-hospital settings where the text comes from different clinical settings than the ones used for training. It enables automated disease annotation across a broad range of clinical diagnoses with minimal preprocessing. The technical framework in this work can be applied in other medical domains that currently lack medical coding resources.
BibTeX
@article{nie2018deeptag,
  author = {Allen Nie and Ashley Zehnder and Rodney López Page and Yuhui Zhang and Arturo López Pineda and Manuel A. Rivas and Carlos D. Bustamante and James Zou},
  title = {{DeepTag: inferring diagnoses from veterinary clinical notes}},
  journal = {Npj Digit. Medicine},
  year = {2018},
  doi = {10.1038/s41746-018-0067-8},
}
Figure