Weakly Supervised Online Action Detection for Infant General Movements

Tongyi Luo, Jia Xiao, Chuncao Zhang, Siheng Chen, Yuan Tian, Guangjun Yu, Kang Dang, Xiaowei Ding

TL;DR

This paper introduces WO-GMA, a novel weakly supervised online action detection method for identifying "fidgety movements" (FMs) in infants, a crucial indicator for early detection of cerebral palsy (CP). By leveraging infant body keypoints and a unique two-branch architecture, WO-GMA can accurately classify and localize FMs in real-time, significantly reducing diagnosis time.

What Problem Does This Paper Address?

Early diagnosis of cerebral palsy (CP) in infants is critical for timely intervention and improving quality of life. General Movements Assessment (GMA) is a highly accurate tool for this purpose, but it's labor-intensive and requires expert interpretation of video recordings. Existing automated GMA methods often require complete video observation for classification and struggle to pinpoint the exact moments when FMs occur. This limitation hinders real-world application where faster, more interpretable diagnoses are needed. The paper addresses this by proposing a method that can detect FMs in an online fashion (processing video as it comes in) and localize them within the video stream, even with only weak supervision (video-level labels instead of precise frame-level annotations).

Methods

The proposed WO-GMA model operates on sequences of 2D infant body keypoints extracted from video. Its architecture comprises three main components:

  1. Local Feature Extraction Module (LFEM): This module uses a spatio-temporal graph convolutional network (GCN) to capture complex local interactions between infant joints over time. It processes segments of the skeleton sequence to extract rich spatio-temporal features.
  2. Clip-level Pseudo Labels Generating Branch (CPGB): This branch is crucial for weakly supervised learning. It leverages both local features from LFEM and long-range temporal information to generate "pseudo labels" for short video clips. These pseudo labels, derived from video-level ground truth, guide the learning process without requiring explicit frame-level annotations. It uses a two-stage thresholding strategy to refine these pseudo labels.
  3. Online Action Modeling Branch (OAMB): This branch uses an LSTM to process the features and pseudo labels generated by the previous modules. It learns to detect FMs in an online manner, meaning it makes predictions based only on past and current information, without looking into the future. This branch also incorporates a Multiple Instance Learning (MIL) loss for video-level classification.

The training process jointly optimizes these components using a combination of MIL losses and frame-level losses derived from the pseudo labels. For inference, only the feature extraction and online action modeling branches are used.

Datasets

The study utilizes a newly collected dataset comprising 757 videos of infants aged 46-70 weeks. This dataset was collected from three hospitals and includes videos of both healthy infants (F+ for fidgety movements) and those at risk (F- for non-fidgety movements). The videos vary in resolution and duration, with an average duration of 307 seconds. The dataset was split into training and testing sets with an 8:2 ratio.

Key Insights and Ideas

  • Skeleton-based input is key: The paper strongly suggests that motion information, captured by skeleton keypoints, is more critical for FMs detection than appearance-based features (like RGB frames or optical flow). This is a significant finding for this specific task.
  • Weak supervision is sufficient: The success of WO-GMA demonstrates that accurate FMs detection and localization can be achieved even with only video-level labels, alleviating the burden of expensive frame-level annotation.
  • Online detection drastically reduces diagnosis time: A major practical implication is that WO-GMA can achieve near-complete video-level classification accuracy using only the first 20% of a video's duration. This could revolutionize the speed of infant neuromotor assessment.
  • Pseudo-labeling bridges the gap: The CPGB's ability to generate reliable clip-level pseudo labels from video-level supervision is a core innovation that enables effective online action detection.

Key Results

  • State-of-the-art performance: WO-GMA achieves state-of-the-art results in both video-level classification (Accuracy: 94.8%, F1: 95.1%, AUC: 96.6%) and clip-level detection (mean Average Precision (mAP) at IoU=0.5: 17.7%) on their custom dataset.
  • Significant speedup: The model achieves comparable video-level classification accuracy using only the first 20% of the video, implying a substantial reduction in diagnosis time.
  • Ablation studies confirm module importance: Removing any of the key components (pseudo-labeling, local feature extraction, or long-range information capture) leads to a noticeable drop in performance, highlighting the synergistic nature of the proposed architecture.

Limitations and Future Work

  • Detection difficulty: The paper acknowledges that action detection is inherently more challenging than classification. The performance gap between video-level classification and clip-level detection suggests that more sophisticated methods might be needed for precise temporal localization.
  • Intermittent FMs: The authors note that the difficulty in accurately determining FMs boundaries for annotators is a challenge. This might also affect the model's ability to precisely segment continuous FMs.
  • Future work: While not explicitly detailed, the paper's success in online detection with weak supervision opens avenues for further research into more efficient and interpretable automated GMA systems. Exploring more advanced temporal modeling techniques or incorporating multi-modal data could be potential future directions.

Summary updated

Abstract

To make the earlier medical intervention of infants’ cerebral palsy (CP), early diagnosis of brain damage is critical. Although general movements assessment (GMA) has shown promising results in early CP detection, it is laborious. Most existing works take videos as input to make fidgety movements (FMs) classification for the GMA automation. Those methods require a complete observation of videos and can not localize video frames containing normal FMs. Therefore we propose a novel approach named WO-GMA to perform FMs localization in the weakly supervised online setting. Infant body keypoints are first extracted as the inputs to WO-GMA. Then WO-GMA performs local spatio-temporal extraction followed by two network branches to generate pseudo clip labels and model online actions. With the clip-level pseudo labels, the action modeling branch learns to detect FMs in an online fashion. Experimental results on a dataset with 757 videos of different infants show that WO-GMA can get state-of-the-art video-level classification and clip-level detection results. Moreover, only the first 20% duration of the video is needed to get classification results as good as fully observed, implying a significantly shortened FMs diagnosis time. Code is available at: https://github.com/scofiedluo/WO-GMA .
BibTeX
@inproceedings{luo2022weakly,
  author = {Tongyi Luo and Jia Xiao and Chuncao Zhang and Siheng Chen and Yuan Tian and Guangjun Yu and Kang Dang and Xiaowei Ding},
  title = {{Weakly Supervised Online Action Detection for Infant General Movements}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=_68T9VlcmdZ},
}
Figure