TL;DR
This paper introduces SV-data2vec, a novel self-supervised learning framework that uses skeleton data to guide the learning of video representations. By pretraining on both modalities and then fine-tuning on video alone, it achieves state-of-the-art performance without requiring explicit pose estimation during inference, simplifying the process and improving efficiency.
What Problem Does This Paper Address?
Action recognition is crucial for understanding human behavior in various applications. While combining video and skeleton data has led to impressive results, existing methods often struggle with effective fusion. Early fusion can underutilize individual modality strengths, while late fusion leads to complex architectures. Furthermore, self-supervised learning for multimodal action recognition is an underexplored area. This paper aims to bridge this gap by developing a self-supervised method that leverages the complementary nature of video and skeleton data to learn robust video representations, even when only video is available for downstream tasks.
Methods
The core of SV-data2vec is a student-teacher architecture, inspired by the data2vec framework.
- Teacher Network: This network processes unmasked skeleton data to generate contextualized "latent skeleton targets." The decision to use only skeleton data for the teacher is based on its inherent clarity and robustness to background noise, making it a reliable guide.
- Student Network: This network receives masked skeleton and video inputs. It performs a masked prediction task, aiming to reconstruct the latent targets generated by the teacher.
- Input Encoding: Both skeleton and video data are encoded using transformer-based encoders. For video, a modified ResNet-18 is used as the initial feature extractor.
- Modality Fusion (Student): For the student network, if both skeleton and video tokens are visible after masking, their features are averaged. If only video tokens are visible, they are used directly.
- Masking: A random masking strategy is employed, with a higher masking ratio for skeletons (90%) than for video (70%). This encourages the model to rely on video information when skeleton data is absent. Crucially, if a video token is masked, its corresponding skeleton token is also masked to prevent the model from learning from skeleton data alone.
- Self-supervised Pretraining: The student decoder attempts to reconstruct the teacher's latent representations using Mean Squared Error (MSE) loss. The teacher's weights are updated via an Exponential Moving Average (EMA) of the student's weights, a common technique to stabilize training.
- Fine-tuning: After pretraining, the student encoder's weights are extracted, a classification head is added, and the model is fine-tuned for action classification using only RGB video data.
Datasets
The paper evaluates SV-data2vec on three benchmark datasets:
- NTU RGB+D 60 (NTU 60): A standard benchmark for human action recognition with 60 action classes.
- NTU RGB+D 120 (NTU 120): An extension of NTU 60 with an additional 60 action classes, totaling 120.
- Toyota Smarthome: A dataset for recognizing daily living activities, with 31 classes.
For NTU RGB+D datasets, 2D skeletons estimated by HRNet are used. For the Toyota Smarthome dataset, 2D skeletons provided by the dataset are utilized.
Key Insights and Ideas
- Skeleton as a "Teacher": The core idea of using skeleton data as a guiding signal for learning video representations is powerful. Skeletons provide explicit motion cues that are less susceptible to background clutter, making them ideal for generating robust targets.
- Self-supervised Learning for Multimodal Fusion: This work demonstrates the effectiveness of self-supervised learning in bridging the gap between modalities, simplifying model design and reducing reliance on labeled data for pretraining.
- Inference without Pose Estimation: The ability to fine-tune and perform inference using only RGB video, achieving competitive results, is a significant practical advantage. This bypasses the often computationally expensive and error-prone step of skeleton extraction during deployment.
- Masking Strategy Matters: The asymmetric masking strategy (higher for skeletons) is crucial. It forces the model to learn from video when skeleton information is missing, thereby learning richer video representations.
Key Results
- State-of-the-Art Performance: When fine-tuned with RGB data alone, SV-data2vec achieves performance on par with or exceeding existing multimodal approaches on NTU 60 and NTU 120 datasets.
- Improved RGB-only Performance: The method shows significant improvements (2.6% and 2.4% on NTU 60 and NTU 120 CS protocols, respectively) compared to previous RGB-only state-of-the-art methods when fine-tuned with only video input.
- Robustness to Noisy Skeletons: The approach demonstrates competitive performance even when using noisier 2D skeletons derived from projected 3D data, highlighting its robustness.
- Challenging Action Pairs: SV-data2vec significantly improves performance on visually similar action pairs that are difficult to distinguish using RGB data alone.
Limitations and Future Work
- Fully Transformer Architecture: The authors suggest transitioning to a fully transformer-based architecture for both encoders and decoders to improve generality and scalability, eliminating the need for a separate visual convolutional encoder.
- Advanced Fusion Techniques: Exploring more advanced fusion techniques for multimodal fine-tuning is a potential avenue for further performance gains.
- Broader Applicability: The authors plan to apply SV-data2vec to other tasks and datasets to assess its robustness and adaptability.
- Target Generation Modality: While using skeleton-only targets for the teacher proved most effective, the paper notes that pretraining with combined RGB-skeleton targets also converges but results in a slight performance drop, likely due to RGB noise. Pretraining with RGB-only targets did not converge.
Summary updated