Automated Coding of Under-Studied Medical Concept Domains: Linking Physical Activity Reports to the International Classification of Functioning, Disability, and Health

Denis Newman-Griffis, Eric Fosler-Lussier

TL;DR

This paper introduces a framework for automatically mapping clinical "activity reports" (e.g., "patient walked 300ft with a cane") to standardized codes in the International Classification of Functioning, Disability, and Health (ICF). By comparing classification and candidate selection paradigms, the authors achieve a macro-averaged F1 score of 84%, demonstrating that high-performing medical NLP is possible even in "under-studied" domains with limited annotated data.

What Problem Does This Paper Address?

While medical NLP has made massive strides in identifying diseases and medications, it often ignores Functional Status Information (FSI)—the data describing how a patient actually moves, communicates, and interacts with their environment.

FSI is critical for rehabilitation, disability claims, and elderly care, but it’s notoriously difficult to automate. The standard coding system, the ICF, is a classification (a hierarchy of concepts) rather than a terminology (a dictionary of how people actually talk). Consequently, there is a massive "semantic gap" between a physical therapist's notes and the formal ICF codes. Furthermore, because this domain is under-studied, there aren't massive, multi-million-sample annotated datasets available to train standard deep learning models.

Methods

The authors investigated two primary NLP paradigms to bridge this gap:

  1. Classification: Treating ICF codes as fixed categories. They tested established models like k-Nearest Neighbors (KNN), Support Vector Machines (SVM), and Deep Neural Networks (DNN) using both unigram features and word embeddings.
  2. Candidate Selection: A more flexible approach where the model compares the mathematical representation of a clinical note to the mathematical representation of an ICF code's definition.
  3. Novelty: They introduced a context-dependent projection model. Instead of just comparing static vectors, a DNN learns to "project" code definitions into a space that better matches the context of the specific activity report being analyzed.

They also experimented with an "Action Oracle"—a setting where the model is told exactly which word in the sentence represents the physical action (e.g., "walking"). This helped determine how much performance is lost simply because the model can't distinguish the "action" from the "context" (like the assistive device used).

Datasets

  • Primary Dataset: 400 Physical Therapy records from the NIH Clinical Center, containing 4,527 annotated activity reports mapped to 12 distinct ICF mobility codes (plus an "Other" category).
  • Pre-training Corpora: To test the impact of domain-specific language, they used several datasets for word embeddings:
  • General: Google News (100B words).
  • Clinical: MIMIC-III (ICU notes).
  • Institutional: NIHCC Large (general NIH notes).
  • Domain-Specific: PT-OT (Physical/Occupational Therapy notes from NIH).
  • Contextual Embeddings: BERT-Base, BioBERT, and clinicalBERT.

Key Insights and Ideas

  • Representativeness Trumps Size: In the world of word embeddings, "more" isn't always "better." The tiny PT-OT corpus (1.2M words) consistently outperformed the massive Google News corpus (100B words) because it captured the specific jargon and shorthand used by therapists.
  • The "Action" is the Signal: Knowing exactly which word describes the action (the Action Oracle) boosted performance by 12-20%. This suggests that future work should focus heavily on "Action Extraction" as a pre-processing step.
  • Classification vs. Flexibility: While classification models yielded the highest raw scores, the authors argue that candidate selection is more promising for the long term. Why? Because it can handle new codes or domains without needing to retrain the entire model from scratch—you just need a written definition of the new code.
  • The "Other" Problem: Candidate selection struggles with the "open world" problem. If a patient's activity doesn't fit any known code (the "Other" label), a similarity-based model will still try to force it into the "closest" (but incorrect) category.

Key Results

  • Top Performance: The best system (SVM using domain-specific PT-OT embeddings) achieved a macro-averaged F1 score of 84% when given access to the Action Oracle.
  • Without the Oracle: Performance dropped to around 64-65%, highlighting the difficulty of identifying the core action within a complex clinical sentence.
  • BERT Performance: Interestingly, standard SVMs with domain-specific static embeddings often outperformed or matched complex BERT-based models, likely due to the small size of the annotated dataset.

Limitations and Future Work

  • Single-Site Bias: All data came from the NIH Clinical Center. Clinical documentation styles vary wildly between hospitals, so these models need to be tested on "outside" data.
  • The "Other" Label: The candidate selection models currently cannot handle the "Other" category effectively because "Other" has no formal definition to map to.
  • Joint Modeling: The authors suggest that the next big step is "jointly modeling" extraction and coding—essentially teaching the model to find the activity report and code it in one single, cohesive step rather than two separate ones.

Summary updated

Abstract

Linking clinical narratives to standardized vocabularies and coding systems is a key component of unlocking the information in medical text for analysis. However, many domains of medical concepts, such as functional outcomes and social determinants of health, lack well-developed terminologies that can support effective coding of medical text. We present a framework for developing natural language processing (NLP) technologies for automated coding of medical information in under-studied domains, and demonstrate its applicability through a case study on physical mobility function. Mobility function is a component of many health measures, from post-acute care and surgical outcomes to chronic frailty and disability, and is represented as one domain of human activity in the International Classification of Functioning, Disability, and Health (ICF). However, mobility and other types of functional activity remain under-studied in the medical informatics literature, and neither the ICF nor commonly-used medical terminologies capture functional status terminology in practice. We investigated two data-driven paradigms, classification and candidate selection, to link narrative observations of mobility status to standardized ICF codes, using a dataset of clinical narratives from physical therapy encounters. Recent advances in language modeling and word embedding were used as features for established machine learning models and a novel deep learning approach, achieving a macro-averaged ...
BibTeX
@inproceedings{newmangriffis2021automated,
  author = {Denis Newman-Griffis and Eric Fosler-Lussier},
  title = {{Automated Coding of Under-Studied Medical Concept Domains: Linking Physical Activity Reports to the International Classification of Functioning, Disability, and Health}},
  booktitle = {Frontiers Digit. Health},
  year = {2021},
  url = {https://openreview.net/forum?id=7NsfIMtZO2},
}

Loading related content...

Figure