Abstract
Recognizing Activities of Daily Living (ADL) is a critical yet challenging task in computer vision, characterized by fine-grained action differences and significant viewpoint variability. We introduce PI-ViT (Pose Induced Video Transformer), a framework designed to enhance RGB-based video transformers by "inducing" 2D and 3D pose information during the training phase. By utilizing auxiliary tasks to teach the model about human skeletal dynamics, PI-ViT learns viewpoint-agnostic and motion-sensitive representations. Crucially, these pose-related modules are discarded at inference, allowing the model to achieve state-of-the-art accuracy with zero additional computational overhead or reliance on depth sensors at runtime.
Introduction: The Challenges of ADL Recognition
Unlike general action recognition (e.g., "running" vs. "jumping"), understanding Activities of Daily Living (ADL) requires distinguishing between highly similar interactions. For instance, "pouring from a kettle" and "pouring from a bottle" share nearly identical backgrounds and global motion patterns, differing only in subtle hand-object interactions. Furthermore, in real-world home environments, cameras are often placed at varying angles, introducing significant viewpoint variability that can confuse standard appearance-based models.
While human pose data (skeletons) is inherently robust to viewpoint changes and captures fine-grained motion effectively, obtaining high-quality 3D poses at runtime is often impractical due to the latency of pose estimation models or the need for specialized hardware.
The PI-ViT Framework
The core innovation of PI-ViT is the use of pose information as a "teacher" during training rather than a required input during inference. The framework integrates two primary "plug-in" modules into a standard video transformer backbone:
- 2D Skeleton Induction Module (2D-SIM): This module facilitates spatial grounding by training the model to predict a "Token-Skeleton Map." This forces the transformer to attend to specific visual patches containing key human joints.
- 3D Skeleton Induction Module (3D-SIM): This module focuses on viewpoint invariance. It aligns high-level RGB features with features from a pre-trained 3D skeleton model (e.g., Hyperformer) through "hallucination" tasks, such as predicting 3D skeletal features from RGB tokens.
A key finding in the development of PI-ViT was that the placement of these modules matters: 2D-SIM is most effective in early layers for spatial guidance, while 3D-SIM provides the most benefit in later layers for high-level action representation.
Experimental Results and Insights
The authors evaluated PI-ViT on three major benchmarks: Toyota Smarthome (SH), NTU RGB+D 120, and NTU RGB+D 60.
Performance and Efficiency
On the Toyota Smarthome dataset, PI-ViT achieved a mean Class Accuracy (mCA) of 72.9%, representing a 6.6% improvement over the TimeSformer baseline. Notably, PI-ViT outperformed several multi-modal models (like VPN++ and PoseC3D) that require both RGB and 3D pose data at inference time. Because the induction modules are removed after training, PI-ViT maintains the same inference speed as the baseline RGB transformer (approx. 1.5 seconds per video).
Robustness and Generalization
Ablation studies revealed that the 3D-SIM module acts as a powerful regularizer, making the model robust even when training skeletons are noisy. Furthermore, the combination of 2D-SIM (knowing where to look) and 3D-SIM (knowing how the body moves) proved complementary, yielding the highest performance across all protocols.
Conclusion
PI-ViT demonstrates that pose information can be effectively distilled into RGB representations to solve the unique challenges of ADL recognition. By shifting the "pose tax" from inference to training, the framework provides a high-accuracy, viewpoint-robust solution suitable for real-time applications in smart home environments. Future work may explore self-supervised methods for pose induction to reduce reliance on pre-trained skeletal models.
Summary updated