Self-Supervised Learning for Multilevel Skeleton-Based Forgery Detection via Temporal-Causal Consistency of Actions

Liang Hu, Dora D. Liu, Qi Zhang, Usman Naseem, Zhong Yuan LaiAAAI 2023 · OpenReview

Abstract

Skeleton-based human action recognition is vital for security and anomaly detection. However, tampering attacks on skeletal features have emerged as a significant threat. This paper proposes a self-supervised learning approach focused on the temporal causality of human actions to detect temporal inconsistency and incoherence (TII) in skeletal sequences. We introduce a multilevel forgery detection framework targeting frame, clip, and action levels. By employing a hierarchical graph convolution network (GCN) architecture, the model learns low-level physical connections and high-level temporal-causal dependencies. Experimental results on NTU-RGB+D and PKU-MMD datasets demonstrate state-of-the-art performance in multilevel forgery detection.

Introduction

As skeleton-based action recognition becomes ubiquitous in surveillance and autonomous systems, the risk of skeletal forgery grows. Unlike pixel-based deepfakes, skeletal forgery manipulates 3D joint coordinates. Such tampering can mislead critical systems—for instance, a self-driving car might misinterpret a pedestrian's intent if their skeletal data is altered from "walking" to "standing." Existing models primarily focus on static physical joint connections, often neglecting the temporal causality inherent in human movement—how one joint's motion precedes and influences another's. This work addresses this gap by introducing a framework dedicated to Skeleton-based Forgery Detection (SFD) through the lens of temporal-causal consistency.

Multilevel skeleton-based forgery types: frame-level, clip-level, and action-level.

Methodology

The proposed Temporal-Causal SFD Network (TC-SFDN) utilizes a hierarchical architecture to capture both physical and behavioral nuances. The system is composed of three primary modules:

  1. Physical Residual GCN (P-ResGCN): This layer learns representations based on the static physical topology of the human skeleton, ensuring the basic structural integrity of the model.
  2. Causal Residual GCN (C-ResGCN): This module models action-specific causal dependencies. It uses Pseudo Transfer Entropy (pTE) to construct a "Temporal-Causal Graph," identifying joint influences over time (e.g., shoulder movement preceding wrist movement in a "throw").
  3. Temporal Reduce ConvNet (TRCN): This component aggregates information across the temporal sequence to form high-level action clip embeddings.

To train the model without massive labeled forgery datasets, a self-supervised learning (SSL) strategy is employed. Tampered samples are generated at three levels: swapping individual frames (frame-level), disordering frame sequences (clip-level), and replacing segments with different actions (action-level).

Architecture of the TC-SFDN framework, featuring P-ResGCN, C-ResGCN, and TRCN modules for multilevel forgery detection.

Results and Discussion

The framework was evaluated on the NTU-RGB+D and PKU-MMD datasets. TC-SFDN significantly outperformed baseline motion prediction and anomaly detection models like ResRNN and MPED-RNN across all levels:

  • Next Frame Prediction: The model achieved 0.3634 accuracy for the "drink water" category, nearly doubling the performance of the best baseline (0.1992).
  • Clip-Level Detection: For detecting disordered frames, the model reached an Area Under Curve (AUC) of 0.8527 for "drinking water" and 0.8977 for "back pain," whereas baselines struggled around 0.50.
  • Action-Level Sensitivity: Even when only 25% of a clip was tampered with, the model maintained high detection rates (AUC > 0.76), proving sensitive to partial forgeries.

These results underscore that causality is action-specific; modeling these dependencies allows the network to detect subtle biological motion violations that standard physical models miss.

Conclusion

TC-SFDN establishes a robust baseline for skeleton-based forgery detection by leveraging temporal-causal consistency. While effective, future work must address the potential for more sophisticated generative forgeries and optimize computational efficiency for real-time applications. Beyond security, these temporal-causal representations hold promise for improving human-robot interaction and imitation learning.

Summary updated

Abstract

Skeleton-based human action recognition and analysis have become increasingly attainable in many areas, such as security surveillance and anomaly detection. Given the prevalence of skeleton-based applications, tampering attacks on human skeletal features have emerged very recently. In particular, checking the temporal inconsistency and/or incoherence (TII) in the skeletal sequence of human action is a principle of forgery detection. To this end, we propose an approach to self-supervised learning of the temporal causality behind human action, which can effectively check TII in skeletal sequences. Especially, we design a multilevel skeleton-based forgery detection framework to recognize the forgery on frame level, clip level, and action level in terms of learning the corresponding temporal-causal skeleton representations for each level. Specifically, a hierarchical graph convolution network architecture is designed to learn low-level skeleton representations based on physical skeleton connections and high-level action representations based on temporal-causal dependencies for specific actions. Extensive experiments consistently show state-of-the-art results on multilevel forgery detection tasks and superior performance of our framework compared to current competing methods.
BibTeX
@inproceedings{hu2023self,
  author = {Liang Hu and Dora D. Liu and Qi Zhang and Usman Naseem and Zhong Yuan Lai},
  title = {{Self-Supervised Learning for Multilevel Skeleton-Based Forgery Detection via Temporal-Causal Consistency of Actions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=H3srWbV4rX},
}
Figure