Abstract
Video Action Detection (VAD) entails localizing and categorizing action instances within videos, which inherently consist of diverse information sources such as audio, visual cues, and surrounding scene contexts. Leveraging this multi-modal information effectively for VAD poses a significant challenge, as the model must identify action-relevant cues with precision. In this study, we introduce a novel multi-modal VAD architecture, referred to as the Joint Actor-centric Visual, Audio, Language Encoder (JoVALE). JoVALE is the first VAD method to integrate audio and visual features with scene descriptive context sourced from large-capacity image captioning models. At the heart of JoVALE is the actor-centric aggregation of audio, visual, and scene descriptive information, enabling adaptive integration of crucial features for recognizing each actor's actions. We have developed a Transformer-based architecture, the Actor-centric Multi-modal Fusion Network, specifically designed to capture the dynamic interactions among actors and their multi-modal contexts. Our evaluation on three prominent VAD benchmarks, including AVA, UCF101-24, and JHMDB51-21, demonstrates that incorporating multi-modal information significantly enhances performance, setting new state-of-the-art performances in the field.
Introduction
Video Action Detection (VAD) is a complex task that requires both spatial localization (bounding boxes) and temporal classification of human actions. While humans naturally integrate visual, auditory, and contextual information to interpret actions, most existing VAD models rely predominantly on visual data. The challenge in multi-modal VAD lies in the "messiness" of contextual cues; for instance, a specific sound might be highly relevant to one actor's action but merely noise for another in the same scene. JoVALE addresses this by introducing an actor-centric fusion approach that selectively routes multi-modal information based on individual actor contexts.
The JoVALE Framework
JoVALE (Joint Actor-centric Visual, Audio, Language Encoder) utilizes a multi-stage pipeline to process video data. It begins with a person detector (e.g., Faster R-CNN) to identify actors in keyframes. Subsequently, three distinct types of embeddings are extracted:
- Visual Embeddings: Derived from standard video backbones like SlowFast or ViT.
- Audio Embeddings: Generated from log-mel-spectrograms of the accompanying audio.
- Language Embeddings: A novel addition where a Vision-Language Pre-training (VLP) model (BLIP) generates text descriptions of the scene, providing high-level semantic context.
Actor-centric Multi-modal Fusion Network (AMFN)
The core of JoVALE is the AMFN, which avoids simple feature concatenation in favor of a more sophisticated adaptive integration. The AMFN consists of two primary components:
- Multi-modal Feature Encoding (MFE): This module processes temporal dynamics. It employs "Temporal Bottleneck Features" to efficiently compress information across actors over time, managing computational complexity.
- Multi-modal Feature Aggregation (MFA): This module implements an adaptive gated fusion mechanism. It calculates dynamic weights for each modality (video, audio, text) on a per-actor basis, allowing the model to prioritize the most relevant information for a specific action.
Experimental Evaluation
The model was evaluated on three major benchmarks:
- AVA v2.2: A challenging dataset for spatio-temporally localized atomic actions with rich audio-visual content.
- UCF101-24 and JHMDB51-21: Datasets primarily focused on visual actions, used to test the synergy between visual and language features in the absence of audio.
Key Results and Insights
JoVALE achieved state-of-the-art performance across all benchmarks. On AVA v2.2, it reached a 40.1% mAP using a ViT-B backbone, a significant improvement over previous methods. Even on datasets lacking audio, the combination of visual and language features set new records (84.9% on UCF101-24 and 91.0% on JHMDB51-21).
Ablation studies highlighted the necessity of actor-centric routing. Simple multi-modal fusion yielded 31.9 mAP, whereas the adaptive gated fusion improved this to 34.0 mAP. Furthermore, while visual data remains the most critical modality (28.0 mAP alone), the addition of text (32.7 mAP) and audio (34.0 mAP) demonstrates strong complementarity. Qualitative analysis showed that multi-modal cues help the model focus its attention more accurately on action-relevant objects, such as musical instruments when corresponding sounds are present.
Conclusion
JoVALE demonstrates that integrating language-based scene context with traditional audio-visual features significantly enhances video action detection. By employing an actor-centric fusion strategy, the model effectively manages the complexity of multi-modal data. While the computational cost of running multiple large encoders is a current limitation, future work exploring generic, pre-trained Vision-Language Models (VLMs) with advanced prompting could further refine contextual understanding and efficiency.
Summary updated