Novel AI driven approach to classify infant motor functions

Simon Reich, Dajie Zhang, Tomas Kulvičius, Sven Bölte, Karin Nielsen‐Saines, Florian B. Pokorny, Robert Peharz, Luise Poustka, Florentin Wörgötter, Christa Einspieler, Peter B. MarschikScientific 2021 · DOI

TL;DR

This paper introduces a novel AI-driven approach to automatically classify "fidgety movements" (FMs) in infants, a key indicator of neurodevelopmental health. By using a single RGB camera and the OpenPose framework to extract infant skeleton data, a shallow neural network achieves an impressive 88% classification accuracy, paving the way for more accessible and efficient infant motor function assessments.

What Problem Does This Paper Address?

Assessing infant motor functions is crucial for early detection of neurological disorders like cerebral palsy. The gold standard, the Prechtl General Movements Assessment (GMA), relies on expert visual interpretation of spontaneous infant movements. While highly effective, GMA requires extensive training and is susceptible to human factors like fatigue and subjectivity. This limits its widespread accessibility, especially in resource-constrained settings. The paper aims to bridge this gap by developing an AI system that can automate the detection of specific, age-appropriate movement patterns, like fidgety movements (FMs), which are strong predictors of normal neurodevelopment. FMs, characterized by small amplitude, moderate speed movements of the neck, trunk, and limbs, typically emerge around 3-5 months postterm and are predictive of normal neurodevelopment. Their absence at this age is a strong indicator of potential neurological impairments.

Methods

The core of this research lies in a novel pipeline that combines computer vision with a shallow neural network for movement classification.

  1. Data Acquisition: Infants were recorded using a single, passive RGB camera during their natural movements.
  2. Pose Estimation: The OpenPose framework was employed to extract a 25-point skeleton representation of the infant from each video frame. This is a key advantage as it avoids the need for markers on the infant or complex multi-camera setups.
  3. Feature Extraction: The (x, y) coordinates of these 25 skeleton points were normalized and stacked across multiple frames to create an input vector for the neural network. This "normalized skeleton" serves as the feature representation.
  4. Classification: A Shallow Multilayer Neural Network (SMNN) was designed and trained to classify these feature vectors into "fidgety movements" (FM+) or "non-fidgety movements" (FM-). The authors explored various SMNN architectures, varying the number of hidden layers and neurons, and performed an ablation study to optimize hyperparameters like batch size, number of stacked frames ($N_{stack}$), and slide offset ($N_{slide}$).

Datasets

The study utilized a prospectively collected dataset of 51 typically developing infants.

  • Data Collection: Infants were recorded bi-weekly from 4 to 16 weeks postterm.
  • Annotation: 2800 five-second video snippets were randomly selected and annotated by two experienced GMA assessors. These snippets were categorized as "FM+" (fidgety movements present) or "FM-" (fidgety movements absent). Inter- and intra-rater reliability was excellent (Cohen's kappa $\kappa = 0.97$ and $\kappa = 0.85-0.95$, respectively).
  • Dataset Splitting: The annotated data was split into training, validation, and testing sets using a cross-validation approach (5 sets) to ensure robust evaluation. A total of 1784 snippets were used for machine learning after excluding "not assessable" or discrepantly labeled segments.

Key Insights and Ideas

  • Skeleton as a Powerful Feature: The paper demonstrates that a normalized 25-point skeleton representation, derived from readily available RGB video, is a sufficiently rich feature set for classifying complex infant movements. This bypasses the need for more complex sensor data or intricate feature engineering.
  • Shallow Networks Suffice: Contrary to the trend towards deep, complex architectures, the authors show that a relatively shallow neural network (SMNN) can achieve high accuracy for this specific task. This is attributed to the well-structured, low-dimensional feature vector derived from the skeleton.
  • Efficiency and Accessibility: The proposed method relies on a single RGB camera, making it highly accessible and potentially deployable in various clinical and even home settings. The short training and inference times further enhance its practical utility.
  • Focus on Age-Specific Patterns: Unlike some prior work that focused on differentiating typical from abnormal movements, this study specifically targets the detection of an age-specific typical movement pattern (FMs), highlighting a nuanced approach to developmental assessment.

Key Results

  • Classification Accuracy: The SMNN achieved an average classification accuracy of 88% for discriminating fidgety from non-fidgety movements.
  • Hyperparameter Optimization: An ablation study identified optimal hyperparameters: $N_{batch} = 3968$, $N_{stack} = 52$ frames (representing ~1.04 seconds of video), and $N_{slide} = 12$ frames.
  • Network Performance: Among the nine SMNN architectures tested, SMNN 5 (with two hidden layers) demonstrated the most stable performance across cross-validation sets, with a slight statistical advantage over SMNN 4.
  • Training and Inference Speed: The SMNNs exhibited fast training times (4.2-8.5 minutes per training set) and very rapid inference times (approximately 19 ms per sample).

Limitations and Future Work

  • Dataset Size: While the dataset is well-annotated, the authors acknowledge that larger datasets are always beneficial for machine learning models.
  • Home Recordings: The authors suggest investigating the feasibility of using home-recorded videos for automated GMA in the future. This presents a significant challenge due to the heterogeneity of home environments and recording conditions.
  • Clinical Reasoning Augmentation: The paper emphasizes that AI tools are intended to augment, not replace, human clinical reasoning. The goal is to provide a consistent, accurate, and efficient screening tool that complements expert assessment.
  • Generalizability: While the study focused on typically developing infants, future work could explore the model's performance on diverse clinical populations and different age ranges.

Summary updated

Abstract

The past decade has evinced a boom of computer-based approaches to aid movement assessment in early infancy. Increasing interests have been dedicated to develop AI driven approaches to complement the classic Prechtl general movements assessment (GMA). This study proposes a novel machine learning algorithm to detect an age-specific movement pattern, the fidgety movements (FMs), in a prospectively collected sample of typically developing infants. Participants were recorded using a passive, single camera RGB video stream. The dataset of 2800 five-second snippets was annotated by two well-trained and experienced GMA assessors, with excellent inter- and intra-rater reliabilities. Using OpenPose, the infant full pose was recovered from the video stream in the form of a 25-points skeleton. This skeleton was used as input vector for a shallow multilayer neural network (SMNN). An ablation study was performed to justify the network's architecture and hyperparameters. We show for the first time that the SMNN is sufficient to discriminate fidgety from non-fidgety movements in a sample of age-specific typical movements with a classification accuracy of 88%. The computer-based solutions will complement original GMA to consistently perform accurate and efficient screening and diagnosis that may become universally accessible in daily clinical practice in the future.
BibTeX
@inproceedings{reich2021novel,
  author = {Simon Reich and Dajie Zhang and Tomas Kulvičius and Sven Bölte and Karin Nielsen‐Saines and Florian B. Pokorny and Robert Peharz and Luise Poustka and Florentin Wörgötter and Christa Einspieler and Peter B. Marschik},
  title = {{Novel AI driven approach to classify infant motor functions}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-89347-5},
}
Figure