Unified Framework with Consistency across Modalities for Human Activity Recognition

Tuyen Tran, Thao Minh Le, Hung Tran, Truyen TranarXiv 2024 · DOI · arXiv

Introduction

Recognizing human activities in video is a complex challenge for artificial intelligence. It requires a deep understanding of individual movements, interactions between people, and the broader environmental context over time. Traditionally, researchers have relied on two distinct data modalities: RGB Video, which provides rich contextual information but is computationally intensive, and Skeleton Data (Keypoints), which is efficient for motion but lacks scene context.

Combining these modalities—often referred to as the "holy grail" of activity recognition—is difficult because they are fundamentally different data types. Simple late-fusion techniques frequently lead to one modality "polluting" the other, resulting in suboptimal performance. To address this, we introduce COMPUTER (COMPositional hUman-cenTric quERy machine), a unified neural architecture designed to harmonize these disparate data sources.

The COMPUTER Architecture

Overview of the COMPUTER framework, illustrating the input modalities, the modular HUB architecture, and the dual loss functions (L_CC and L_CE).

At the heart of COMPUTER is a modular design called the HUB (HUman-centric query Block). Unlike specialized branches for different data types, the HUB block is modality-agnostic. The framework treats activity recognition as a series of "queries" focused on a specific person.

The architecture operates through a two-stage interaction process:

  1. Human-Human (HH) Interaction: Modeling how an actor moves relative to others in the scene.
  2. Human-Context (HC) Interaction: Understanding how the actor interacts with their environment.

To capture temporal dynamics, each HUB block incorporates a memory mechanism that analyzes "past," "current," and "future" video segments, allowing the model to understand how relationships and actions evolve over time.

Cross-Modality Consistency

A critical innovation in this framework is the Cross-Modality Consistency Loss ($L_{CC}$). This self-supervised mechanism bridges the gap between pixels (RGB) and points (Skeletons). During training, the model is penalized if the RGB representation and the skeletal representation of the same person do not align within the model's internal latent space. This forces the architecture to extract complementary, high-level features from both sources without requiring additional manual labels.

Experimental Results

COMPUTER was evaluated on two major benchmarks: AVA v2.2 for spatio-temporal action localization and the Collective Activity Dataset for group activity recognition.

Quantitative Performance

  • State-of-the-Art on AVA: COMPUTER achieved 33.6 mAP (mean Average Precision), surpassing previous leading models such as HIT and AIA.
  • Group Activity Accuracy: On the Collective Activity dataset, the model reached 93.3% accuracy, a significant improvement over the 91.0% baseline.
  • Efficiency: By focusing on human-centric queries rather than dense pixel processing, the model is remarkably efficient. Using a small backbone (MViTv2-S), COMPUTER outperformed larger backbones while utilizing only one-third of the computational power (GFLOPs).

Qualitative Analysis

Qualitative results demonstrating the model's ability to accurately label complex actions like "Driving" and "Running" by integrating pose and RGB data.

The model demonstrates particular strength in recognizing actions that rely on fine-grained temporal cues. As shown in the qualitative examples, skeletal data provides critical motion cues for actions like "running" or "driving" that might be ambiguous in static RGB frames alone. Ablation studies confirmed that removing the consistency loss results in a 1.5-point drop in performance, highlighting the importance of modality alignment.

Limitations and Future Work

Despite its performance, COMPUTER currently relies on external pre-processing for human bounding boxes and skeletal keypoints. If the initial detector fails, the system cannot recognize the activity. Future research will explore end-to-end integration to eliminate this dependency and investigate the inclusion of additional modalities, such as audio or infrared data, into the unified query framework.

Summary updated

Abstract

Recognizing human activities in videos is challenging due to the spatio-temporal complexity and context-dependence of human interactions. Prior studies often rely on single input modalities, such as RGB or skeletal data, limiting their ability to exploit the complementary advantages across modalities. Recent studies focus on combining these two modalities using simple feature fusion techniques. However, due to the inherent disparities in representation between these input modalities, designing a unified neural network architecture to effectively leverage their complementary information remains a significant challenge. To address this, we propose a comprehensive multimodal framework for robust video-based human activity recognition. Our key contribution is the introduction of a novel compositional query machine, called COMPUTER ($\textbf{COMP}ositional h\textbf{U}man-cen\textbf{T}ric qu\textbf{ER}y$ machine), a generic neural architecture that models the interactions between a human of interest and its surroundings in both space and time. Thanks to its versatile design, COMPUTER can be leveraged to distill distinctive representations for various input modalities. Additionally, we introduce a consistency loss that enforces agreement in prediction between modalities, exploiting the complementary information from multimodal inputs for robust human movement recognition. Through extensive experiments on action localization and group activity recognition tasks, our approach demonstrates superior performance when compared with state-of-the-art methods. Our code is available at: https://github.com/tranxuantuyen/COMPUTER.
BibTeX
@misc{tran2024unified,
  author = {Tuyen Tran and Thao Minh Le and Hung Tran and Truyen Tran},
  title = {{Unified Framework with Consistency across Modalities for Human Activity Recognition}},
  year = {2024},
  eprint = {2409.02385},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.02385},
}
Figure