Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection

Wentao Bao, Kai Li, Yuxiao Chen, Deep Patel, Martin Renqiang Min, Yu Kong

Abstract

Action detection aims to detect (recognize and localize) human actions spatially and temporally in videos. Existing approaches focus on the closed-set setting where an action detector is trained and tested on videos from a fixed set of action categories. However, this constrained setting is not viable in an open world where test videos inevitably come beyond the trained action categories. In this paper, we address the practical yet challenging Open-Vocabulary Action Detection (OVAD) problem. It aims to detect any action in test videos while training a model on a fixed set of action categories. To achieve such an open-vocabulary capability, we propose a novel method OpenMixer that exploits the inherent semantics and localizability of large vision-language models (VLM) within the family of query-based detection transformers (DETR). Specifically, the OpenMixer is developed by spatial and temporal OpenMixer blocks (S-OMB and T-OMB), and a dynamically fused alignment (DFA) module. The three components collectively enjoy the merits of strong generalization from pre-trained VLMs and end-to-end learning from DETR design. Moreover, we established OVAD benchmarks under various settings, and the experimental results show that the OpenMixer performs the best over baselines for detecting seen and unseen actions.

Introduction

Traditional action detection models operate in a "closed-set" environment. If a model is trained to recognize "walking" and "running," it remains blind to "jumping" or "dancing" in real-world scenarios. This limitation is a significant bottleneck for practical applications like autonomous driving or smart surveillance, where the variety of human actions is nearly infinite.

The challenge of Open-Vocabulary Action Detection (OVAD) is twofold:

  1. Localization: Finding a person performing an action that the model has never encountered during training.
  2. Semantics: Mapping complex, moving visual patterns to a text-based concept (e.g., "throwing a ball") without specific training examples.

While Vision-Language Models (VLMs) like CLIP have revolutionized image understanding, they often struggle with the temporal "verbs" of video and lack the fine-grained spatial awareness needed to pinpoint exactly where an action is occurring. OpenMixer addresses these gaps by repurposing the internal attention and semantic knowledge of pre-trained VLMs.

Methodology

OpenMixer is a query-based detector built on the DETR (Detection Transformer) architecture. It consists of three innovative components designed to extract and utilize knowledge from frozen VLMs.

The OpenMixer framework integrates VLM features into a series of OpenMixer Blocks to produce action scores, bounding boxes, and person scores.

Core Components

  • Spatial OpenMixer Block (S-OMB): Instead of starting with random guesses for person locations, S-OMB uses the VLM’s internal visual attention as a "prior." It identifies regions where video patches correlate strongly with text descriptions, effectively using the VLM as a built-in person proposal generator.
  • Temporal OpenMixer Block (T-OMB): To handle motion, T-OMB employs "Adaptive Semantics." It takes global video-level features from the VLM and injects them into the model's temporal queries before they interact with the video data. This "pre-conditioning" helps the model understand the overall context (e.g., "this is a sports video") before decoding specific movements.
  • Dynamically Fused Alignment (DFA): This module aligns detected video regions with text embeddings. To improve accuracy, the authors used GPT-4 to generate rich, descriptive prompts for each action. DFA then uses a learnable fusion mechanism to balance general VLM knowledge with the specific features learned during training.
Detailed view of the S-OMB, T-OMB, and DFA modules, showing how queries are mixed with visual features and conditioned on text semantics.

Key Insights

  • Reversed Attention as a Spatial Prior: The authors noted that VLM attention often highlights the background rather than the foreground. By reversing this attention map, they obtained a highly effective spatial prior for human localization.
  • Pre-conditioning vs. Post-conditioning: Injecting VLM semantic features early in the query decoding process (pre-conditioning) is significantly more effective than adding them at the end. It guides the model's search through the video rather than attempting to correct results post-hoc.
  • Decoupling Localization and Recognition: By making the localization part of the model class-agnostic (focusing on "finding a person" regardless of the specific action), the model generalizes much better to unseen categories.

Experiments and Results

The researchers evaluated OpenMixer on two gold-standard action detection datasets, creating new "Open-Vocabulary" splits:

  • J-HMDB: 21 classes, split into 10 "base" (seen) and 11 "novel" (unseen) classes.
  • UCF101-24: 24 classes, split into 12 base and 12 novel classes.

OpenMixer demonstrated a massive leap in performance over existing state-of-the-art methods. On the J-HMDB dataset, OpenMixer achieved 82.33% novel mAP, compared to the STMixer baseline's 27.44%. On UCF101-24, it reached 34.23% compared to the baseline's 6.54%. Even in a "Zero-Shot" setting with no training on the target dataset, OpenMixer outperformed specialized models like iCLIP by over 11% in frame-level mAP. The model maintains a competitive inference speed of 0.23 seconds per video on an A6000 GPU.

Conclusion and Future Work

OpenMixer successfully enables open-vocabulary action detection by leveraging the latent knowledge within VLMs. While highly effective on J-HMDB and UCF101-24, future work will focus on scaling the model to massive, "in-the-wild" datasets like AVA, which feature more complex, multi-person scenes and longer temporal dependencies. The team plans to explore pre-training on these larger datasets to create a more universal action detector.

Summary updated

Abstract

Action detection aims to detect (recognize and localize) human actions spatially and temporally in videos. Existing approaches focus on the closed-set setting where an action detector is trained and tested on videos from a fixed set of action categories. However, this constrained setting is not viable in an open world where test videos inevitably come beyond the trained action categories. In this paper, we address the practical yet challenging Open-Vocabulary Action Detection (OVAD) problem. It aims to detect any action in test videos while training a model on a fixed set of action categories. To achieve such an open-vocabulary capability, we propose a novel method OpenMixer that exploits the inherent semantics and localizability of large vision-language models (VLM) within the family of query-based detection transformers (DETR). Specifically, the OpenMixer is developed by spatial and temporal OpenMixer blocks (S-OMB and T-OMB), and a dynamically fused alignment (DFA) module. The three components collectively enjoy the merits of strong generalization from pre-trained VLMs and end-to-end learning from DETR design. Moreover, we established OVAD benchmarks under various settings, and the experimental results show that the OpenMixer performs the best over baselines for detecting seen and unseen actions. We release the codes, models, and dataset splits at https://github.com/Cogito2012/OpenMixer.
BibTeX
@inproceedings{bao2025exploiting,
  author = {Wentao Bao and Kai Li and Yuxiao Chen and Deep Patel and Martin Renqiang Min and Yu Kong},
  title = {{Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00804},
}
Figure