Cross-Modal Learning with 3D Deformable Attention for Action Recognition

Sangwon Kim, Dasom Ahn, ByoungChul Ko

Introduction

Action recognition is a cornerstone of computer vision, requiring an AI to interpret not just static objects, but their dynamic evolution over time. While 3D Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have pushed the boundaries of this field, they often struggle with high computational costs and rigid receptive fields. This paper introduces a 3D Deformable Transformer designed to efficiently fuse video (RGB) and skeletal (pose) data using adaptive spatiotemporal attention.

The Challenge of Spatiotemporal Complexity

Two primary hurdles hinder current action recognition models:

  1. Modality Fusion: Effectively combining raw video pixels with skeletal joint coordinates without creating redundant or overly complex architectures.
  2. Computational Efficiency: Standard Transformers compare every image patch (token) across every frame. This quadratic complexity makes processing long videos memory-intensive. Furthermore, fixed grid-like receptive fields in standard ViTs often fail to capture the most discriminative parts of an action, such as a specific limb movement or an object interaction.

Proposed Method: 3D Deformable Transformer

The authors propose a leaner, smarter attention mechanism composed of three core innovations:

1. 3D Deformable Attention

Unlike traditional deformable models that operate on 2D spatial offsets, this module searches across both space and time. Using a 3D Token Search (3DTS), the model adaptively identifies the most relevant tokens throughout the video sequence. This allows the "gaze" of the model to follow the action dynamically, rather than being restricted to a static grid.

Comparison between Full Attention and the proposed 3D Deformable Attention, highlighting the efficiency of adaptive token selection.

2. Cross-Modal Learning Scheme

To integrate different data types, the architecture employs a "modality-mixer" token ($M_{CLS}$). This token acts as a bridge within a single Transformer stream, facilitating the exchange of contextual information between RGB and Pose features without requiring separate, parallel models.

3. Stride Attention Modules

To maintain high processing speeds, the model utilizes Local Joint Stride and Temporal Stride attention. By using sliding windows to group tokens, the model captures local correlations (e.g., the relationship between a wrist and an elbow) and temporal shifts without the need for exhaustive global comparisons.

The overall model pipeline (a) and the detailed architecture of the 3D Deformable Attention block (b).

Experimental Evaluation

The model was rigorously tested on four major benchmarks:

  • NTU RGB+D 60 & 120: Large-scale datasets for general human activity.
  • FineGYM: A dataset focused on fine-grained gymnastic maneuvers.
  • PennAction: A collection of complex, high-intensity sports actions.

Key Results

The 3D Deformable Transformer achieved state-of-the-art (SoTA) performance, notably reaching 99.7% Top-1 accuracy on PennAction. On the NTU60 dataset, it achieved 94.3% (Cross-Subject) accuracy. Remarkably, the model outperformed many existing methods that rely on massive pre-training, even when trained from scratch. On the challenging FineGYM dataset, it reached 90.3% mean accuracy, demonstrating its ability to distinguish between very similar athletic motions.

Insights and Discussion

  • Adaptive Receptive Fields: Visualizations confirm that the 3D deformable attention "locks on" to the subject (e.g., a gymnast) while ignoring irrelevant background noise.
  • Efficiency through Overlap: Optimal performance was found when the temporal stride was roughly half the window size, ensuring continuity in action perception.
  • Pre-training is Not Always Mandatory: The efficiency of the deformable and stride mechanisms allows the model to learn complex patterns effectively from smaller datasets.

Limitations and Future Work

The model's performance is sensitive to the number of input frames, with accuracy peaking at 12 frames. Future research may explore scaling the architecture for even longer sequences or applying 3D deformable offsets to tasks like video object detection and tracking. Additionally, further work could enhance the transparency of the cross-modal exchange between RGB and Pose data.

Summary updated

Abstract

An important challenge in vision-based action recognition is the embedding of spatiotemporal features with two or more heterogeneous modalities into a single feature. In this study, we propose a new 3D deformable transformer for action recognition with adaptive spatiotemporal receptive fields and a cross-modal learning scheme. The 3D deformable transformer consists of three attention modules: 3D deformability, local joint stride, and temporal stride attention. The two cross-modal tokens are input into the 3D deformable attention module to create a cross-attention token with a reflected spatiotemporal correlation. Local joint stride attention is applied to spatially combine attention and pose tokens. Temporal stride attention temporally reduces the number of input tokens in the attention module and supports temporal expression learning without the simultaneous use of all tokens. The deformable transformer iterates L-times and combines the last cross-modal token for classification. The proposed 3D deformable transformer was tested on the NTU60, NTU120, FineGYM, and PennAction datasets, and showed results better than or similar to pre-trained state-of-the-art methods even without a pre-training process. In addition, by visualizing important joints and correlations during action recognition through spatial joint and temporal stride attention, the possibility of achieving an explainable potential for action recognition is presented.
BibTeX
@inproceedings{kim2022cross,
  author = {Sangwon Kim and Dasom Ahn and ByoungChul Ko},
  title = {{Cross-Modal Learning with 3D Deformable Attention for Action Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2022},
}
Figure