Classifying the reported ability in clinical mobility descriptions

Denis Newman-Griffis, Ayah Zirikly, Guy Divita, Bart Desmet

TL;DR

This paper tackles the challenge of automatically understanding how well patients can perform mobility-related activities based on clinical notes. They developed and evaluated machine learning models, including an ensemble approach, that achieve a strong baseline performance in classifying these descriptions into "Able," "Unable," "Unclear," or "None" categories, with their best model reaching 77.9% macro F1 score. This work is crucial for better modeling patient health and informing public policy.

What Problem Does This Paper Address?

Understanding a patient's ability to perform daily activities is fundamental for assessing their health status, quality of life, and disability. This information is particularly critical for mobility, as it's highly dependent on the environment. For instance, someone might be able to walk easily in an office but struggle on a mountain path. Clinical notes often contain rich descriptions of these activities, but they are complex. They can include subtle syntactic negations, require understanding context, and may not always provide a clear assertion of ability. Extracting this nuanced information automatically is a significant challenge. This paper addresses the gap of systematically classifying these "activity performance assertions" within clinical text, a task that goes beyond simple negation detection by considering the context and environment.

Methods

The authors explored a variety of methods, starting with a rule-based approach using the UIMA framework and the conTEXT algorithm, which leverages a lexicon of evidence and clues to assign polarity. They then moved to machine learning, evaluating several common algorithms: Random Forest, Naïve Bayes, k-Nearest Neighbors, Support Vector Machines (SVM), and Deep Neural Networks (DNN). For feature representation, they experimented with both traditional binary unigram features and modern word embeddings (FastText).

A key aspect of their work is the development of a Convolutional Neural Network (CNN) architecture specifically designed for this task. This CNN combines word and character embeddings to handle out-of-vocabulary words and leverages context surrounding the action mention. Finally, they investigated ensemble methods, combining the predictions of their best individual models (SVM and CNN) using majority voting and a DNN chooser, to further improve performance.

Datasets

The study utilized an extended version of a dataset initially described by Thieu et al. (2017). This dataset consists of 400 English-language Physical Therapy initial assessment and reassessment notes from the Rehabilitation Medicine Department of the NIH Clinical Center. These notes were manually annotated to identify mobility descriptions and assign one of four "Polarity" labels: "Able," "Unable," "Unclear," or "None." The dataset was split into training (320 documents) and testing (80 documents) sets, stratified by Polarity label distribution.

Key Insights and Ideas

  • Nuance of Negation: The paper highlights that simple negation detection isn't enough. For example, "Patient can walk with rolling walker" and "Patient cannot walk without rolling walker" both assert the ability to walk with assistance, demonstrating the need to consider context and assistive devices.
  • Importance of Context: Activity performance is not absolute but highly context-dependent. The models need to understand the environmental factors mentioned in the text to accurately classify ability.
  • Rarity of Negative Assertions: The "Unable" and "Unclear" classes are significantly rarer than "Able" and "None." This class imbalance poses a challenge for machine learning models and requires careful evaluation and potentially specialized techniques.
  • Ensembling for Robustness: Combining multiple models, particularly an ensemble of SVM and CNN, proved effective in striking a balance between high recall (catching rare cases) and high precision (accuracy), leading to the best overall performance.
  • CNNs for Text Classification: The CNN architecture, by combining word and character embeddings, showed promise, especially in handling the rare "Unable" class, outperforming other systems in some metrics.

Key Results

The best-performing model was an ensemble using majority voting, achieving a macro F1 score of 77.9%. The DNN chooser ensemble also performed strongly with 77.4% macro F1. Individual models showed competitive results, with SVM achieving 81.7% macro F1 and the CNN reaching 76.4% macro F1 in cross-validation experiments. Notably, the ensemble methods were able to achieve high recall for the rare "Unable" and "Unclear" classes while maintaining good precision, a significant improvement over individual models that often traded one for the other. The CNN model, in particular, achieved the highest F1 score for the "Unable" class on the test set.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Negation Scoping: The rule-based system's negation scoping was limited to sentence boundaries, and a more granular approach (e.g., dependency parsing) is needed.
  • Contextual Ambiguity: The "Unclear" label is inherently ambiguous, even for humans, making it a difficult class to classify.
  • Hypothetical Statements: Distinguishing between factual statements of ability (e.g., "Pt can run") and hypothetical or aspirational ones (e.g., "Pt eager to walk") is crucial and requires further model development.
  • Assistance Mentions: Explicitly modeling mentions of assistance and assistive devices is needed to better capture nuanced ability descriptions.
  • Generalizability: The dataset is from a single specialty (Physical Therapy) at one institution. Future work should evaluate generalization to broader clinical subdomains and different institutions.
  • Advanced Embeddings: Incorporating contextual embeddings like ELMo and BERT could further enhance performance.
  • Hybrid Models: Investigating hybrid models that combine lexical features with information about report formatting, section structure, and assistive devices is a promising avenue.

Summary updated

Abstract

Assessing how individuals perform different activities is key information for modeling health states of individuals and populations. Descriptions of activity performance in clinical free text are complex, including syntactic negation and similarities to textual entailment tasks. We explore a variety of methods for the novel task of classifying four types of assertions about activity performance: Able, Unable, Unclear, and None (no information). We find that ensembling an SVM trained with lexical features and a CNN achieves 77.9% macro F1 score on our task, and yields nearly 80% recall on the rare Unclear and Unable samples. Finally, we highlight several challenges in classifying performance assertions, including capturing information about sources of assistance, incorporating syntactic structure and negation scope, and handling new modalities at test time. Our findings establish a strong baseline for this novel task, and identify intriguing areas for further research.
BibTeX
@inproceedings{newmangriffis2019classifying,
  author = {Denis Newman-Griffis and Ayah Zirikly and Guy Divita and Bart Desmet},
  title = {{Classifying the reported ability in clinical mobility descriptions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure