Two-stream Multi-level Dynamic Point Transformer for Two-person Interaction Recognition

Yao Liu, Gangfeng Cui, Jiahui Luo, Xiaojun Chang, Lina Yao

TL;DR

This paper introduces the Two-stream Multi-level Dynamic Point Transformer (TMDPT), a novel framework for recognizing interactions between two people using 3D point clouds derived from depth videos. By combining a smart frame-sampling strategy with a transformer-based architecture that looks at both global and partial temporal segments, the model achieves state-of-the-art accuracy while maintaining high levels of personal privacy.

What Problem Does This Paper Address?

Recognizing how two people interact (e.g., hugging, punching, shaking hands) is significantly harder than recognizing single-person actions. The primary challenges are body occlusion and overlap, where one person hides parts of the other from the camera's view.

Existing methods often fall into two traps:

  1. Skeleton-based methods: These rely on pre-defined joints. When people overlap, joint estimation becomes error-prone, leading to "garbage in, garbage out" scenarios.
  2. RGB-based methods: These raise significant privacy concerns, as they capture identifiable facial and environmental textures.

The authors propose using point clouds from depth sensors. Point clouds provide rich 3D geometric information without the privacy baggage of RGB video and are more robust to occlusion than skeleton models because they represent the raw surface data of the subjects.

Methods

The TMDPT architecture consists of four main stages:

  1. Interval Frame Sampling (IFS): Instead of simple uniform sampling, IFS uses a two-step process. It first selects a large pool of frames (Top frame rate) and then, during each training epoch, randomly samples a smaller subset (Bottom frame rate). This acts as a form of temporal data augmentation, forcing the network to learn from different slices of the interaction.
  2. Frame Features Learning Module: For every sampled frame, the model uses Set Abstraction (SA) layers (similar to PointNet++) and a Convolutional Block Attention Module (CBAM) to extract three things: local-region features, frame spatial features, and frame temporal features (using sinusoidal positional encoding).
  3. Two-stream Multi-level Feature Aggregation: This is the core "aha" of the architecture. It processes the video in two streams:
  4. Global Stream: Aggregates features across the entire video.
  5. Partial Stream: Splits the video into temporal segments to capture fine-grained motion details that might get "washed out" in a global view.
  6. Transformer Classification: The aggregated global and partial features are fed into a Transformer. The self-attention mechanism allows the global features to "attend" to specific partial details and vice versa, creating a comprehensive representation for final classification.

Datasets

The model was evaluated on the interaction subsets of two massive benchmarks:

  • NTU RGB+D 60: Contains 11 interaction classes and over 10,000 video samples.
  • NTU RGB+D 120: An extension of the above, featuring 26 interaction classes and nearly 25,000 samples. It is currently the largest dataset for 3D action recognition.

Key Insights and Ideas

  • The "Double Advantage" of IFS: The authors found that their sampling method not only reduces redundancy (saving computation) but also provides a "positive data augmentation-like effect." Because the specific frames change slightly every epoch, the model becomes more robust.
  • Privacy by Design: By converting depth maps to point clouds, the model ignores color and texture, making it suitable for sensitive environments like hospitals or smart homes.
  • Global vs. Local Context: The two-stream approach acknowledges that an interaction like "shaking hands" requires both the global context (two people standing near each other) and fine-grained partial motion (the specific movement of the hands over a few frames).

Key Results

The TMDPT model outperformed almost all existing state-of-the-art (SOTA) methods, including complex skeleton-based and hybrid models:

  • NTU RGB+D 60: Achieved 96.6% accuracy in Cross-Subject (X-Sub) and 99.0% in Cross-View (X-View) settings.
  • NTU RGB+D 120: Achieved 92.7% (X-Sub) and 93.3% (Cross-Setup).
  • Efficiency: Despite using Transformers, the model is remarkably efficient. It requires only 7.09 billion FLOPs, compared to 54.80 billion for the previous leading point-cloud model (SequentialPointNet), while being nearly 3x faster in relative processing time.

Limitations and Future Work

While the paper demonstrates excellent results, a few areas remain for exploration:

  • Depth Sensor Dependency: The model relies on high-quality depth data. Its performance might degrade with noisy data from low-cost sensors or in outdoor environments where infrared-based depth sensing struggles.
  • Fixed Segment Count: The partial stream uses a fixed number of temporal segments (set to 6 in experiments). Future work could investigate dynamic segmenting based on the complexity or duration of the action.
  • Extreme Occlusion: While more robust than skeletons, total occlusion (one person completely behind another) still poses a fundamental geometric challenge for any single-view system.

Summary updated

Abstract

As a fundamental aspect of human life, two-person interactions contain meaningful information about people's activities, relationships, and social settings. Human action recognition serves as the foundation for many smart applications, with a strong focus on personal privacy. However, recognizing two-person interactions poses more challenges due to increased body occlusion and overlap compared to single-person actions. In this paper, we propose a point cloud-based network named Two-stream Multi-level Dynamic Point Transformer for two-person interaction recognition. Our model addresses the challenge of recognizing two-person interactions by incorporating local-region spatial information, appearance information, and motion information. To achieve this, we introduce a designed frame selection method named Interval Frame Sampling (IFS), which efficiently samples frames from videos, capturing more discriminative information in a relatively short processing time. Subsequently, a frame features learning module and a two-stream multi-level feature aggregation module extract global and partial features from the sampled frames, effectively representing the local-region spatial information, appearance information, and motion information related to the interactions. Finally, we apply a transformer to perform self-attention on the learned features for the final classification. Extensive experiments are conducted on two large-scale datasets, the interaction subsets of NTU RGB+D 60 and NTU RGB+D 120. The results show that our network outperforms state-of-the-art approaches in most standard evaluation settings.
BibTeX
@article{liu2024two,
  author = {Yao Liu and Gangfeng Cui and Jiahui Luo and Xiaojun Chang and Lina Yao},
  title = {{Two-stream Multi-level Dynamic Point Transformer for Two-person Interaction Recognition}},
  journal = {ACM Trans. Multim. Comput. Commun. Appl.},
  year = {2024},
  doi = {10.1145/3639470},
}
Figure