Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection

Wentao Bao, Kai Li, Yuxiao Chen, Deep Patel, Martin Renqiang Min, Yu Kong

Standard spatio-temporal action detection (AD) has traditionally operated in a closed-set environment, where models are limited to recognizing only the specific action categories present in their training data. This limitation hinders the deployment of such models in real-world scenarios where novel actions frequently occur. To address this, researchers have introduced OpenMixer, a framework designed for Open-Vocabulary Action Detection (OVAD). Presented at WACV, OpenMixer leverages the latent knowledge within large-scale Vision-Language Models (VLMs) to detect and localize human actions it has never explicitly seen during training.

The OpenMixer Framework

OpenMixer is built upon the Detection Transformer (DETR) architecture, adapted for the complexities of video data. The core challenge of moving to an open-vocabulary setting is twofold: effectively localizing a person performing an unknown action and understanding the specific motion dynamics of that action using general knowledge from models like CLIP, which are primarily trained on static images.

The OpenMixer architecture integrates video VLM features with query-based detection, featuring the internal OpenMixer Block structure.

As shown in the architecture overview, OpenMixer processes video and text through a frozen VLM to extract visual context ($V$), location priors ($S$), and semantic features ($f_t, f_v$). These features are then fed into a series of OpenMixer Blocks that refine temporal and spatial queries to produce final bounding boxes and action scores.

Core Components

The innovation of OpenMixer lies in its three primary modules, which decouple person localization from action recognition to preserve generalization capabilities.

Detailed mechanisms of the Spatial OpenMixer Block (S-OMB), Temporal OpenMixer Block (T-OMB), and Dynamically Fused Alignment (DFA) module.

1. Spatial OpenMixer Block (S-OMB)

Instead of relying on random initialization for actor localization, S-OMB utilizes VLM Attention. By calculating the correlation between video patches and text features, the model generates a spatial heatmap. This provides a robust spatial prior that helps the model converge on actors even for unseen classes. The module uses location sampling and Q-V (Query-Visual) mixing to refine spatial queries ($Q_s$) into accurate bounding boxes.

2. Temporal OpenMixer Block (T-OMB)

To capture motion dynamics, T-OMB employs Adaptive Semantics. It uses the global visual feature ($f_v$) from the VLM as a condition to guide temporal queries ($Q_t$). This conditioning allows the model to decode the specific dynamics of the video content before classification, proving more effective than using fixed text features alone.

3. Dynamically Fused Alignment (DFA)

The DFA module serves as the final bridge, dynamically fusing learnable queries with VLM features. It aligns these fused representations with high-quality text prompts—often generated via GPT-4 to provide descriptive context (e.g., "a person swinging a golf club" rather than just "golfing"). This alignment ensures that the model can utilize the VLM's broad semantic knowledge to label actions accurately.

Evaluation and Key Insights

The framework was evaluated on new OVAD benchmarks derived from the J-HMDB and UCF101-24 datasets. The results demonstrate a significant performance leap over traditional baselines:

  • Superior Generalization: On the J-HMDB dataset, OpenMixer achieved a video mAP of 82.33% on novel (unseen) classes, a massive improvement over the STMixer baseline's 27.44%.
  • Seen Class Stability: The model also improved performance on seen classes, reaching 90.75% compared to 73.06% for the baseline.
  • VLMs as Localizers: A key finding is that the internal attention maps of models like CLIP can be repurposed as effective person detectors without additional training.
  • Decoupling is Essential: By separating "finding a person" from "identifying the action," the model avoids overfitting to training categories and maintains its ability to generalize to the "open world."

Limitations and Future Work

While OpenMixer excels in controlled datasets, performance on more complex datasets like UCF101-24 (34.23% mAP on novel classes) highlights ongoing challenges. The longer videos, heavy background bias, and multi-person scenarios in such datasets suggest that temporal modeling in cluttered environments remains difficult. Furthermore, because the model relies on frozen VLMs, its upper bound is currently limited by the resolution and inherent biases of those foundation models. Future research may explore multi-person interaction modeling and scaling the approach to larger, more diverse video datasets.

Summary updated

Abstract

Action detection aims to detect (recognize and localize) human actions spatially and temporally in videos. Existing approaches focus on the closed-set setting where an action detector is trained and tested on videos from a fixed set of action categories. However, this constrained setting is not viable in an open world where test videos inevitably come beyond the trained action categories. In this paper, we address the practical yet challenging Open-Vocabulary Action Detection (OVAD) problem. It aims to detect any action in test videos while training a model on a fixed set of action categories. To achieve such an open-vocabulary capability, we propose a novel method OpenMixer that exploits the inherent semantics and localizability of large vision-language models (VLM) within the family of query-based detection transformers (DETR). Specifically, the OpenMixer is developed by spatial and temporal OpertMixer blocks (S-OMB and T-OMB), and a dynamically fused alignment (DFA) module. The three components collectively enjoy the merits of strong generalization from pretrained VLMs and end-to-end learning from DETR design. Moreover, we established OVAD benchmarks under various settings, and the experimental results show that the OpenMixer performs the best over baselines for detecting seen and unseen actions. We release the codes, models, and dataset splits at https://github.com/Cogito2012/0penMixer.
BibTeX
@inproceedings{bao2025exploiting,
  author = {Wentao Bao and Kai Li and Yuxiao Chen and Deep Patel and Martin Renqiang Min and Yu Kong},
  title = {{Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00804},
}
Figure