JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts

Taein Son, Soo Won Seo, Jisong Kim, Seok Hwan Lee, Jun Won Choi

TL;DR

JoVALE (Joint Actor-centric Visual, Audio, Language Encoder) is a state-of-the-art Video Action Detection (VAD) framework that achieves superior performance by selectively fusing audio, visual, and language-based scene descriptions. By focusing on "actor-centric" aggregation, the model precisely identifies which cues—like the sound of a piano or a text description of a room—are relevant to a specific person's actions, setting new benchmarks on the AVA, UCF101-24, and JHMDB51-21 datasets.

What Problem Does This Paper Address?

Video Action Detection (VAD) is significantly more complex than simple action recognition. While recognition asks "what is happening in this clip?", VAD asks "who is doing what, where, and when?" This requires localizing actors with bounding boxes and classifying their actions across time.

The challenge lies in the "noise" of multi-modal data. In a crowded scene, the sound of a gunshot is highly relevant to a person "firing a weapon," but irrelevant to someone "running" in the background of the same shot. Existing methods often struggle to link these disparate cues (audio, pixels, and scene context) to the correct actor. Furthermore, while vision-language models have revolutionized image understanding, their potential for providing rich, semantic scene context in the VAD pipeline has remained largely untapped.

Methods

The authors introduce JoVALE, a framework built around the Actor-centric Multi-modal Fusion Network (AMFN).

Figure 1: Overview of JoVALE: (top-left) The proposed JoVALE integrates audio, visual, and scene-descriptive features using an AMFN. (bottom-left) JoVALE leverages a VLP model fine-tuned on an image captioning task to generate scene-descriptive features. (right) AMFN encodes high-level interactions between multi-modal features through MFE and MFA.

The architecture consists of three primary stages:

  1. Feature Extraction: The model extracts standard visual and audio features. Uniquely, it also uses a pre-trained BLIP (Bootstrapping Language-Image Pre-training) model to generate "scene-descriptive" features—essentially high-level semantic "captions" of the environment.
  2. Multi-modal Feature Encoding (MFE): To keep the Transformer-based architecture efficient, the authors use "Temporal Bottleneck Features." These compress information across all actors at each time step, allowing the model to capture temporal dynamics without a massive computational overhead.
  3. Multi-modal Feature Aggregation (MFA): This is the "actor-centric" heart of the model. It uses an adaptive gated fusion mechanism to decide how much weight to give to the audio, visual, or language features for each specific actor proposal.
Figure 3: Structure of Multi-modal Feature Encoding. This illustration depicts the process for the visual modality. Iden- tical structures are applied individually to other modalities.

Datasets

The model was evaluated on three major VAD benchmarks:

  • AVA (v2.2): A large-scale dataset of 15-minute movie clips with 60 action classes.
  • UCF101-24: A subset of the UCF101 dataset containing 24 sports-related action classes.
  • JHMDB51-21: A subset of the JHMDB dataset with 21 action classes.

Key Insights and Ideas

  • Language as a "Third Eye": By incorporating scene descriptions from a Vision-Language Foundation model, JoVALE gains a semantic understanding of the environment (e.g., "standing in a room," "holding a phone") that raw visual backbones might miss.
  • Selective Attention: The "aha moment" is that multi-modal fusion shouldn't be global. By making fusion actor-centric, the model learns to ignore the "piano music" when classifying a "talking" action, even if both are happening in the same frame.
  • Cross-Modal Synergy: Audio doesn't just provide extra labels; it helps the visual model "focus." For instance, hearing a cello helps the model better localize the instrument in the visual activation maps.
Visualization showing how multi-modal input (V+A+S) improves localization compared to visual-only input.

Key Results

JoVALE achieved state-of-the-art (SOTA) results across the board:

  • AVA Dataset: Recorded a mean Average Precision (mAP) of 40.1%, a substantial 2.4% improvement over the previous best method (EVAD).
  • UCF101-24 & JHMDB51-21: Achieved 84.9% and 91.0% mAP, respectively. Notably, JoVALE reached these SOTA levels on these datasets even though they largely lack audio, proving the strength of its visual and scene-descriptive fusion.
  • Ablation Studies: The authors showed that combining all three modalities (Video + Audio + Scene) outperformed any combination of two, highlighting the complementary nature of these data sources.

Limitations and Future Work

The primary trade-off for JoVALE's performance is computational complexity. Incorporating the BLIP model and processing multiple modalities increases the GFLOPs compared to simpler visual-only models. While the authors demonstrated that reducing BLIP's input resolution can mitigate this with only a slight performance hit, efficiency remains a concern for real-time applications.

Future work suggests exploring more advanced "prompting" strategies with foundation models and integrating even larger, more general-purpose Vision-Language Models (VLMs) to further refine scene understanding.

Summary updated

Abstract

Video Action Detection (VAD) entails localizing and categorizing action instances within videos, which inherently consist of diverse information sources such as audio, visual cues, and surrounding scene contexts. Leveraging this multi-modal information effectively for VAD poses a significant challenge, as the model must identify action-relevant cues with precision. In this study, we introduce a novel multi-modal VAD architecture, referred to as the Joint Actor-centric Visual, Audio, Language Encoder (JoVALE). JoVALE is the first VAD method to integrate audio and visual features with scene descriptive context sourced from large-capacity image captioning models. At the heart of JoVALE is the actor-centric aggregation of audio, visual, and scene descriptive information, enabling adaptive integration of crucial features for recognizing each actor's actions. We have developed a Transformer-based architecture, the Actor-centric Multi-modal Fusion Network, specifically designed to capture the dynamic interactions among actors and their multi-modal contexts. Our evaluation on three prominent VAD benchmarks, including AVA, UCF101-24, and JHMDB51-21, demonstrates that incorporating multi-modal information significantly enhances performance, setting new state-of-the-art performances in the field.
BibTeX
@inproceedings{son2025jovale,
  author = {Taein Son and Soo Won Seo and Jisong Kim and Seok Hwan Lee and Jun Won Choi},
  title = {{JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32745},
}
Figure