Introduction
Recognizing human activities in video is a complex challenge for artificial intelligence. It requires a deep understanding of individual movements, interactions between people, and the broader environmental context over time. Traditionally, researchers have relied on two distinct data modalities: RGB Video, which provides rich contextual information but is computationally intensive, and Skeleton Data (Keypoints), which is efficient for motion but lacks scene context.
Combining these modalities—often referred to as the "holy grail" of activity recognition—is difficult because they are fundamentally different data types. Simple late-fusion techniques frequently lead to one modality "polluting" the other, resulting in suboptimal performance. To address this, we introduce COMPUTER (COMPositional hUman-cenTric quERy machine), a unified neural architecture designed to harmonize these disparate data sources.
The COMPUTER Architecture
At the heart of COMPUTER is a modular design called the HUB (HUman-centric query Block). Unlike specialized branches for different data types, the HUB block is modality-agnostic. The framework treats activity recognition as a series of "queries" focused on a specific person.
The architecture operates through a two-stage interaction process:
- Human-Human (HH) Interaction: Modeling how an actor moves relative to others in the scene.
- Human-Context (HC) Interaction: Understanding how the actor interacts with their environment.
To capture temporal dynamics, each HUB block incorporates a memory mechanism that analyzes "past," "current," and "future" video segments, allowing the model to understand how relationships and actions evolve over time.
Cross-Modality Consistency
A critical innovation in this framework is the Cross-Modality Consistency Loss ($L_{CC}$). This self-supervised mechanism bridges the gap between pixels (RGB) and points (Skeletons). During training, the model is penalized if the RGB representation and the skeletal representation of the same person do not align within the model's internal latent space. This forces the architecture to extract complementary, high-level features from both sources without requiring additional manual labels.
Experimental Results
COMPUTER was evaluated on two major benchmarks: AVA v2.2 for spatio-temporal action localization and the Collective Activity Dataset for group activity recognition.
Quantitative Performance
- State-of-the-Art on AVA: COMPUTER achieved 33.6 mAP (mean Average Precision), surpassing previous leading models such as HIT and AIA.
- Group Activity Accuracy: On the Collective Activity dataset, the model reached 93.3% accuracy, a significant improvement over the 91.0% baseline.
- Efficiency: By focusing on human-centric queries rather than dense pixel processing, the model is remarkably efficient. Using a small backbone (MViTv2-S), COMPUTER outperformed larger backbones while utilizing only one-third of the computational power (GFLOPs).
Qualitative Analysis
The model demonstrates particular strength in recognizing actions that rely on fine-grained temporal cues. As shown in the qualitative examples, skeletal data provides critical motion cues for actions like "running" or "driving" that might be ambiguous in static RGB frames alone. Ablation studies confirmed that removing the consistency loss results in a 1.5-point drop in performance, highlighting the importance of modality alignment.
Limitations and Future Work
Despite its performance, COMPUTER currently relies on external pre-processing for human bounding boxes and skeletal keypoints. If the initial detector fails, the system cannot recognize the activity. Future research will explore end-to-end integration to eliminate this dependency and investigate the inclusion of additional modalities, such as audio or infrared data, into the unified query framework.
Summary updated