Classifying the reported ability in clinical mobility descriptions

Denis Newman-Griffis, Ayah Zirikly, Guy Divita, Bart Desmet

TL;DR

This paper introduces the novel task of automatically classifying how patient mobility is described in clinical notes, moving beyond simple keyword matching to understand complex assertions of ability. By ensembling Support Vector Machines (SVMs) and Convolutional Neural Networks (CNNs), the researchers achieved a 77.9% macro F1 score, demonstrating that machine learning can effectively identify even rare instances where a patient's ability is "Unable" or "Unclear."

What Problem Does This Paper Address?

In clinical settings, a patient's functional status—their ability to perform activities like walking or transferring from a chair—is a critical indicator of health and quality of life. However, this information is usually buried in the free-text "Physical Therapy" (PT) notes of Electronic Health Records (EHRs).

Extracting this information is notoriously difficult because clinical language is flexible and context-dependent. A simple negation doesn't always mean a lack of ability. For example, "Patient cannot walk without a rolling walker" is linguistically negative but clinically asserts that the patient is able to walk with assistance. Furthermore, assertions are often ambiguous (e.g., "the cane makes it difficult to walk") or purely hypothetical (e.g., "Goal: walk 50 feet"). The authors tackle the challenge of classifying these descriptions into four categories: Able, Unable, Unclear, and None (no information provided).

Methods

The researchers compared three main approaches:

  1. Rule-based: They utilized the v3NLP-Framework and the conTEXT algorithm, which uses a lexicon of "clues" to identify negation and uncertainty within a specific window of text. They augmented this with custom rules for "slot:value" constructs (e.g., Transfers: Independent).
  2. Traditional Machine Learning: They tested several baselines, including Random Forests, Naïve Bayes, and SVMs. The SVM, using a combination of unigram features and word embeddings, emerged as the strongest baseline.
  3. Deep Learning (CNN): They adopted a CNN architecture (see Figure 1) that combines word embeddings with character-level embeddings. This hybrid approach is particularly useful in clinical text to handle "out-of-vocabulary" terms—typos or highly specific jargon that a standard word-only model might miss.
Figure 1: CNN architecture for Polarity classification.

Finally, they explored Ensemble methods, combining the strengths of the SVM (which had high recall for rare classes) and the CNN (which had high precision). They tested both a simple majority vote and a "DNN chooser"—a secondary neural network trained specifically to decide which model to trust when they disagree.

Datasets

The study used an annotated corpus of 400 Physical Therapy notes from the NIH Clinical Center. This resulted in 4,527 individual action annotations.

The dataset is highly imbalanced, reflecting real-world clinical documentation:

  • None: 2,262 samples (The most common, often referring to goals or scales).
  • Able: 1,982 samples.
  • Unclear: 206 samples.
  • Unable: 77 samples (The rarest and most difficult to detect).

Key Insights and Ideas

  • Context is King: The researchers found that the words preceding an action mention (e.g., "is unable to...") carry significantly more information about ability than the words following it.
  • Character Embeddings Matter: For the rarest class (Unable), character-level embeddings outperformed word-level embeddings. This suggests that the linguistic markers for inability might be more consistent at the sub-word level or that the model benefits from seeing how "un-" prefixes are used.
  • The "None" Trap: A major challenge is distinguishing between a factual report of current ability and a hypothetical goal. Many "None" cases look like "Able" cases (e.g., "Goal: walk independently"), requiring the model to understand the section of the report it is currently analyzing.
  • Complementary Strengths: The SVM was "braver," yielding higher recall for the rare Unable and Unclear classes, while the CNN was more "cautious," yielding higher precision. Ensembling allowed them to strike a balance between the two.

Key Results

The ensemble models provided the best overall performance. The Majority Voting ensemble achieved a 77.9% macro F1 score.

Quantitatively, the machine learning models significantly outperformed the rule-based baseline (which only managed a 42.1% macro F1). As shown in the confusion matrices (Figure 2), the machine learning approaches were particularly good at avoiding confusion between the "Unable" and "Unclear" categories, though they still occasionally struggled with the "None" class.

Figure 2: Confusion matrices for results on the test set.

Qualitatively, the CNN was best at the critical distinction between Able and Unable, while the SVM was better at picking up the nuances of the Unclear category.

Limitations and Future Work

The authors identify several areas for improvement:

  • Dataset Diversity: The current data comes from a single institution and a single specialty (Physical Therapy). Future work needs to test these models on more heterogeneous data from different hospitals.
  • Structural Awareness: The current models treat text mostly as a "bag of words" or a sequence. Incorporating dependency parsing (understanding the grammatical relationship between words) could help resolve complex negations and long-distance dependencies.
  • Assistive Devices: The presence of a walker or cane completely changes the meaning of "unable to walk." The authors plan to model these environmental factors explicitly in future iterations.
  • Modern Architectures: The study suggests that incorporating contextual representations like BERT or ELMo could further boost performance by better capturing the semantic nuances of clinical prose.

Summary updated

Abstract

Assessing how individuals perform different activities is key information for modeling health states of individuals and populations. Descriptions of activity performance in clinical free text are complex, including syntactic negation and similarities to textual entailment tasks. We explore a variety of methods for the novel task of classifying four types of assertions about activity performance: Able, Unable, Unclear, and None (no information). We find that ensembling an SVM trained with lexical features and a CNN achieves 77.9% macro F1 score on our task, and yields nearly 80% recall on the rare Unclear and Unable samples. Finally, we highlight several challenges in classifying performance assertions, including capturing information about sources of assistance, incorporating syntactic structure and negation scope, and handling new modalities at test time. Our findings establish a strong baseline for this novel task, and identify intriguing areas for further research.
BibTeX
@inproceedings{newmangriffis2019classifying,
  author = {Denis Newman-Griffis and Ayah Zirikly and Guy Divita and Bart Desmet},
  title = {{Classifying the reported ability in clinical mobility descriptions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure