Quantifying Impairment and Disease Severity Using AI Models Trained on Healthy Subjects

Boyang Yu, Aakash Kaku, Kangning Liu, Avinash Parnandi, Emily Fokas, Anita Venkatesan, Natasha Pandit, Rajesh Ranganath, Heidi M. Schambra, Carlos Fernandez-Granda

Abstract

This paper introduces COBRA (COnfidence-Based chaRacterization of Anomalies), a novel framework designed to quantify disease severity without the need for large, expert-labeled datasets of impaired patients. By training AI models to perform clinically meaningful tasks using only data from healthy individuals, the researchers demonstrate that the model's drop in confidence when encountering patient data serves as a highly accurate proxy for disease severity. Validated across stroke recovery and knee osteoarthritis, COBRA achieves strong correlations with clinical gold standards while significantly reducing assessment time.

Introduction

In clinical practice, assessing patient recovery or disease progression typically requires specialized, time-consuming examinations. For instance, a standard stroke recovery assessment can take up to 45 minutes. While automated AI solutions are desirable, supervised learning is often hindered by the "small data" bottleneck: the lack of massive, expert-labeled datasets covering every stage of a disease. Traditional anomaly detection methods often focus on visual reconstruction rather than functional clinical metrics. COBRA addresses this by leveraging readily available data from healthy populations to create an automated, clinically relevant metric.

Methodology

The COBRA framework operates through a two-step process centered on task-specific model confidence:

  1. Task-Specific Training: Instead of training a model to detect disease directly, it is trained to perform a "clinically meaningful task" using only healthy subjects. For stroke assessment, the task is action segmentation (e.g., identifying reaching or transporting movements). For knee osteoarthritis, the task is tissue segmentation in MRI scans.
  2. Confidence Aggregation: When the model encounters data from an impaired patient, its confidence (softmax probability) typically drops because the inputs deviate from the healthy training distribution. COBRA calculates a stable metric by averaging this confidence across an entire session or scan, specifically filtering for clinically relevant data classes (e.g., focusing on active motion rather than rest).
Overview of the COBRA framework applied to both functional movement prediction for stroke and tissue segmentation for knee osteoarthritis.

Experimental Setup and Datasets

The researchers validated COBRA across three distinct data modalities:

  • StrokeRehab Dataset: Utilized wearable IMU sensors and video data from 29 healthy individuals and 51 stroke patients, compared against the Fugl-Meyer Assessment (FMA).
  • OAI-ZIB Dataset: Included 3D MRI scans of 101 healthy knees and 378 knees with osteoarthritis, compared against the Kellgren-Lawrence (KL) grade.

Key Results and Insights

COBRA demonstrated that model confidence does not just indicate an anomaly; it drops proportionally to the severity of the impairment. This allows for a granular quantification of disease state using models that have never seen a "sick" example during training.

  • Stroke Assessment: Achieved a Pearson correlation of $\rho = 0.845$ using wearable sensors and $\rho = 0.746$ using video data.
  • Knee Osteoarthritis: Achieved a correlation of $\rho = -0.644$ with the KL grade, where lower confidence directly mapped to higher disease severity.
Density plots showing the distribution of model confidence scores; healthy individuals exhibit significantly higher and more concentrated confidence than impaired individuals.

The study also highlighted the importance of clinical relevance; including non-motion data (like resting periods) significantly weakened the correlation, proving that the AI must perform a task directly impacted by the pathology.

Discussion and Limitations

While COBRA offers a rapid, automated alternative to manual exams, it remains sensitive to environmental confounders. Factors such as lighting, camera angles, or image quality can cause distribution shifts that the model might mistake for clinical impairment. Future research will focus on automatically identifying and controlling for these non-clinical variables and improving the explainability of the confidence drops to provide more actionable feedback for clinicians.

Summary updated

BibTeX
@article{yu2024quantifying,
  author = {Boyang Yu and Aakash Kaku and Kangning Liu and Avinash Parnandi and Emily Fokas and Anita Venkatesan and Natasha Pandit and Rajesh Ranganath and Heidi M. Schambra and Carlos Fernandez-Granda},
  title = {{Quantifying Impairment and Disease Severity Using AI Models Trained on Healthy Subjects}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-024-01173-x},
}
Figure