Abstract
Skeleton-based human action recognition is vital for security and anomaly detection. However, tampering attacks on skeletal features have emerged as a significant threat. This paper proposes a self-supervised learning approach focused on the temporal causality of human actions to detect temporal inconsistency and incoherence (TII) in skeletal sequences. We introduce a multilevel forgery detection framework targeting frame, clip, and action levels. By employing a hierarchical graph convolution network (GCN) architecture, the model learns low-level physical connections and high-level temporal-causal dependencies. Experimental results on NTU-RGB+D and PKU-MMD datasets demonstrate state-of-the-art performance in multilevel forgery detection.
Introduction
As skeleton-based action recognition becomes ubiquitous in surveillance and autonomous systems, the risk of skeletal forgery grows. Unlike pixel-based deepfakes, skeletal forgery manipulates 3D joint coordinates. Such tampering can mislead critical systems—for instance, a self-driving car might misinterpret a pedestrian's intent if their skeletal data is altered from "walking" to "standing." Existing models primarily focus on static physical joint connections, often neglecting the temporal causality inherent in human movement—how one joint's motion precedes and influences another's. This work addresses this gap by introducing a framework dedicated to Skeleton-based Forgery Detection (SFD) through the lens of temporal-causal consistency.
Methodology
The proposed Temporal-Causal SFD Network (TC-SFDN) utilizes a hierarchical architecture to capture both physical and behavioral nuances. The system is composed of three primary modules:
- Physical Residual GCN (P-ResGCN): This layer learns representations based on the static physical topology of the human skeleton, ensuring the basic structural integrity of the model.
- Causal Residual GCN (C-ResGCN): This module models action-specific causal dependencies. It uses Pseudo Transfer Entropy (pTE) to construct a "Temporal-Causal Graph," identifying joint influences over time (e.g., shoulder movement preceding wrist movement in a "throw").
- Temporal Reduce ConvNet (TRCN): This component aggregates information across the temporal sequence to form high-level action clip embeddings.
To train the model without massive labeled forgery datasets, a self-supervised learning (SSL) strategy is employed. Tampered samples are generated at three levels: swapping individual frames (frame-level), disordering frame sequences (clip-level), and replacing segments with different actions (action-level).
Results and Discussion
The framework was evaluated on the NTU-RGB+D and PKU-MMD datasets. TC-SFDN significantly outperformed baseline motion prediction and anomaly detection models like ResRNN and MPED-RNN across all levels:
- Next Frame Prediction: The model achieved 0.3634 accuracy for the "drink water" category, nearly doubling the performance of the best baseline (0.1992).
- Clip-Level Detection: For detecting disordered frames, the model reached an Area Under Curve (AUC) of 0.8527 for "drinking water" and 0.8977 for "back pain," whereas baselines struggled around 0.50.
- Action-Level Sensitivity: Even when only 25% of a clip was tampered with, the model maintained high detection rates (AUC > 0.76), proving sensitive to partial forgeries.
These results underscore that causality is action-specific; modeling these dependencies allows the network to detect subtle biological motion violations that standard physical models miss.
Conclusion
TC-SFDN establishes a robust baseline for skeleton-based forgery detection by leveraging temporal-causal consistency. While effective, future work must address the potential for more sophisticated generative forgeries and optimize computational efficiency for real-time applications. Beyond security, these temporal-causal representations hold promise for improving human-robot interaction and imitation learning.
Summary updated