TL;DR
This paper introduces WO-GMA, a novel weakly supervised online action detection method for identifying "fidgety movements" (FMs) in infants, a crucial indicator for early detection of cerebral palsy (CP). By leveraging infant body keypoints and a unique two-branch architecture, WO-GMA can accurately classify and localize FMs in real-time, significantly reducing diagnosis time.
What Problem Does This Paper Address?
Early diagnosis of cerebral palsy (CP) in infants is critical for timely intervention and improving quality of life. General Movements Assessment (GMA) is a highly accurate tool for this purpose, but it's labor-intensive and requires expert interpretation of video recordings. Existing automated GMA methods often require complete video observation for classification and struggle to pinpoint the exact moments when FMs occur. This limitation hinders real-world application where faster, more interpretable diagnoses are needed. The paper addresses this by proposing a method that can detect FMs in an online fashion (processing video as it comes in) and localize them within the video stream, even with only weak supervision (video-level labels instead of precise frame-level annotations).
Methods
The proposed WO-GMA model operates on sequences of 2D infant body keypoints extracted from video. Its architecture comprises three main components:
- Local Feature Extraction Module (LFEM): This module uses a spatio-temporal graph convolutional network (GCN) to capture complex local interactions between infant joints over time. It processes segments of the skeleton sequence to extract rich spatio-temporal features.
- Clip-level Pseudo Labels Generating Branch (CPGB): This branch is crucial for weakly supervised learning. It leverages both local features from LFEM and long-range temporal information to generate "pseudo labels" for short video clips. These pseudo labels, derived from video-level ground truth, guide the learning process without requiring explicit frame-level annotations. It uses a two-stage thresholding strategy to refine these pseudo labels.
- Online Action Modeling Branch (OAMB): This branch uses an LSTM to process the features and pseudo labels generated by the previous modules. It learns to detect FMs in an online manner, meaning it makes predictions based only on past and current information, without looking into the future. This branch also incorporates a Multiple Instance Learning (MIL) loss for video-level classification.
The training process jointly optimizes these components using a combination of MIL losses and frame-level losses derived from the pseudo labels. For inference, only the feature extraction and online action modeling branches are used.
Datasets
The study utilizes a newly collected dataset comprising 757 videos of infants aged 46-70 weeks. This dataset was collected from three hospitals and includes videos of both healthy infants (F+ for fidgety movements) and those at risk (F- for non-fidgety movements). The videos vary in resolution and duration, with an average duration of 307 seconds. The dataset was split into training and testing sets with an 8:2 ratio.
Key Insights and Ideas
- Skeleton-based input is key: The paper strongly suggests that motion information, captured by skeleton keypoints, is more critical for FMs detection than appearance-based features (like RGB frames or optical flow). This is a significant finding for this specific task.
- Weak supervision is sufficient: The success of WO-GMA demonstrates that accurate FMs detection and localization can be achieved even with only video-level labels, alleviating the burden of expensive frame-level annotation.
- Online detection drastically reduces diagnosis time: A major practical implication is that WO-GMA can achieve near-complete video-level classification accuracy using only the first 20% of a video's duration. This could revolutionize the speed of infant neuromotor assessment.
- Pseudo-labeling bridges the gap: The CPGB's ability to generate reliable clip-level pseudo labels from video-level supervision is a core innovation that enables effective online action detection.
Key Results
- State-of-the-art performance: WO-GMA achieves state-of-the-art results in both video-level classification (Accuracy: 94.8%, F1: 95.1%, AUC: 96.6%) and clip-level detection (mean Average Precision (mAP) at IoU=0.5: 17.7%) on their custom dataset.
- Significant speedup: The model achieves comparable video-level classification accuracy using only the first 20% of the video, implying a substantial reduction in diagnosis time.
- Ablation studies confirm module importance: Removing any of the key components (pseudo-labeling, local feature extraction, or long-range information capture) leads to a noticeable drop in performance, highlighting the synergistic nature of the proposed architecture.
Limitations and Future Work
- Detection difficulty: The paper acknowledges that action detection is inherently more challenging than classification. The performance gap between video-level classification and clip-level detection suggests that more sophisticated methods might be needed for precise temporal localization.
- Intermittent FMs: The authors note that the difficulty in accurately determining FMs boundaries for annotators is a challenge. This might also affect the model's ability to precisely segment continuous FMs.
- Future work: While not explicitly detailed, the paper's success in online detection with weak supervision opens avenues for further research into more efficient and interpretable automated GMA systems. Exploring more advanced temporal modeling techniques or incorporating multi-modal data could be potential future directions.
Summary updated