Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Living

Dominick Reilly, Srijan Das

Abstract

Recognizing Activities of Daily Living (ADL) is a critical yet challenging task in computer vision, characterized by fine-grained action differences and significant viewpoint variability. We introduce PI-ViT (Pose Induced Video Transformer), a framework designed to enhance RGB-based video transformers by "inducing" 2D and 3D pose information during the training phase. By utilizing auxiliary tasks to teach the model about human skeletal dynamics, PI-ViT learns viewpoint-agnostic and motion-sensitive representations. Crucially, these pose-related modules are discarded at inference, allowing the model to achieve state-of-the-art accuracy with zero additional computational overhead or reliance on depth sensors at runtime.

Introduction: The Challenges of ADL Recognition

Unlike general action recognition (e.g., "running" vs. "jumping"), understanding Activities of Daily Living (ADL) requires distinguishing between highly similar interactions. For instance, "pouring from a kettle" and "pouring from a bottle" share nearly identical backgrounds and global motion patterns, differing only in subtle hand-object interactions. Furthermore, in real-world home environments, cameras are often placed at varying angles, introducing significant viewpoint variability that can confuse standard appearance-based models.

Figure 1: ADL recognition is complicated by visually similar actions (fine-grained appearance) and subtle motion differences (fine-grained motion).

While human pose data (skeletons) is inherently robust to viewpoint changes and captures fine-grained motion effectively, obtaining high-quality 3D poses at runtime is often impractical due to the latency of pose estimation models or the need for specialized hardware.

The PI-ViT Framework

The core innovation of PI-ViT is the use of pose information as a "teacher" during training rather than a required input during inference. The framework integrates two primary "plug-in" modules into a standard video transformer backbone:

  1. 2D Skeleton Induction Module (2D-SIM): This module facilitates spatial grounding by training the model to predict a "Token-Skeleton Map." This forces the transformer to attend to specific visual patches containing key human joints.
  2. 3D Skeleton Induction Module (3D-SIM): This module focuses on viewpoint invariance. It aligns high-level RGB features with features from a pre-trained 3D skeleton model (e.g., Hyperformer) through "hallucination" tasks, such as predicting 3D skeletal features from RGB tokens.
Figure 6: Detailed architecture of PI-ViT showing the placement of induction modules and auxiliary tasks. The modules are used only during training to guide the learning of the Video Transformer blocks.

A key finding in the development of PI-ViT was that the placement of these modules matters: 2D-SIM is most effective in early layers for spatial guidance, while 3D-SIM provides the most benefit in later layers for high-level action representation.

Experimental Results and Insights

The authors evaluated PI-ViT on three major benchmarks: Toyota Smarthome (SH), NTU RGB+D 120, and NTU RGB+D 60.

Performance and Efficiency

On the Toyota Smarthome dataset, PI-ViT achieved a mean Class Accuracy (mCA) of 72.9%, representing a 6.6% improvement over the TimeSformer baseline. Notably, PI-ViT outperformed several multi-modal models (like VPN++ and PoseC3D) that require both RGB and 3D pose data at inference time. Because the induction modules are removed after training, PI-ViT maintains the same inference speed as the baseline RGB transformer (approx. 1.5 seconds per video).

Robustness and Generalization

Ablation studies revealed that the 3D-SIM module acts as a powerful regularizer, making the model robust even when training skeletons are noisy. Furthermore, the combination of 2D-SIM (knowing where to look) and 3D-SIM (knowing how the body moves) proved complementary, yielding the highest performance across all protocols.

Conclusion

PI-ViT demonstrates that pose information can be effectively distilled into RGB representations to solve the unique challenges of ADL recognition. By shifting the "pose tax" from inference to training, the framework provides a high-accuracy, viewpoint-robust solution suitable for real-time applications in smart home environments. Future work may explore self-supervised methods for pose induction to reduce reliance on pre-trained skeletal models.

Summary updated

Abstract

Video transformers have become the de facto standard for human action recognition, yet their exclusive reliance on the RGB modality still limits their adoption in certain domains. One such domain is Activities of Daily Living (ADL), where RGB alone is not sufficient to distinguish between visually similar actions, or actions observed from multiple viewpoints. To facilitate the adoption of video transformers for ADL, we hypothesize that the augmentation of RGB with human pose information, known for its sensitivity to fine-grained motion and multiple viewpoints, is essential. Consequently, we introduce the first Pose Induced Video Transformer: PI-ViT (or $π$-ViT), a novel approach that augments the RGB representations learned by video transformers with 2D and 3D pose information. The key elements of $π$-ViT are two plug-in modules, 2D Skeleton Induction Module and 3D Skeleton Induction Module, that are responsible for inducing 2D and 3D pose information into the RGB representations. These modules operate by performing pose-aware auxiliary tasks, a design choice that allows $π$-ViT to discard the modules during inference. Notably, $π$-ViT achieves the state-of-the-art performance on three prominent ADL datasets, encompassing both real-world and large-scale RGB-D datasets, without requiring poses or additional computational overhead at inference.
BibTeX
@inproceedings{reilly2024just,
  author = {Dominick Reilly and Srijan Das},
  title = {{Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Living}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01736},
}
Figure