Semantic embedding space for zero-shot action recognition

Xun Xu, Timothy M. Hospedales, Shaogang Gong

Abstract

This paper presents a scalable framework for Zero-Shot Action Recognition (ZSR) that bypasses the need for manual attribute annotation. By mapping video features into a semantic word vector space (e.g., word2vec), the proposed method bridges the gap between visual data and high-level concepts. To address the inherent domain shift between training and unseen test classes, the authors introduce transductive self-training and cross-dataset augmentation. Experimental results on HMDB51 and UCF101 datasets demonstrate that this approach significantly outperforms traditional attribute-based methods, establishing a new state-of-the-art for zero-shot learning in complex video datasets.

Introduction

Traditional action recognition models are constrained by their dependence on large-scale labeled datasets. As the variety of human actions grows, collecting hundreds of examples for every new category becomes impractical. Zero-Shot Learning (ZSL) offers a solution by enabling models to recognize categories never seen during training.

While previous ZSL research relied heavily on human-defined attributes (e.g., "outdoors," "moving legs"), these are expensive to scale. This work explores the use of Semantic Embedding Spaces (SES)—automatically generated word vectors—to connect raw video pixels to conceptual labels. The primary challenge lies in the "semantic gap": the difficulty of mapping noisy, high-dimensional spatio-temporal video features to a clean mathematical text space.

Methodology

The proposed pipeline consists of three core components designed to create a robust visual-to-semantic bridge:

  1. Semantic Embedding: A skip-gram neural network (word2vec), pre-trained on a 100-billion-word Google News corpus, maps action names into 300-dimensional vectors.
  2. Visual-to-Semantic Mapping: A non-linear Support Vector Regressor (SVR) with an RBF-$\chi^2$ kernel is trained to project low-level video features (Dense Trajectories) into the semantic word vector space.
  3. Ameliorating Domain Shift:
    • Transductive Self-Training: To correct projections that fall "off-target" due to disjoint training and testing classes, the model adjusts class prototypes by averaging the $K$-nearest unlabeled test projections.
  4. Data Augmentation: The regressor is trained on auxiliary datasets to learn a more generalized mapping from video-space to word-space.
Overview of the proposed Zero-Shot Action Recognition pipeline, from input data to semantic mapping and final task execution.

Experimental Setup and Results

The method was evaluated on two rigorous benchmarks: HMDB51 (6,766 videos, 51 categories) and UCF101 (13,320 videos, 101 categories). The authors utilized a 50/50 category split and averaged results over 30 random trials to ensure statistical robustness.

Key Findings

  • Superiority over Attributes: While basic embedding approaches perform similarly to attribute-based methods, the addition of self-training makes the semantic embedding approach significantly more effective.
  • Impact of Self-Training: By analyzing the distribution of unlabeled test data, the model successfully corrects systematic biases introduced by the training set. t-SNE visualizations demonstrate that self-training effectively moves class prototypes into the centers of their respective projected video clusters.
  • Cross-Dataset Synergy: Training the visual-to-semantic regressor on auxiliary datasets (e.g., using UCF101 to improve HMDB51 performance) suggests that the mapping captures fundamental relationships that transcend specific dataset boundaries.
t-SNE visualization showing how self-training (+ST, stars) shifts class prototypes closer to the actual projected data clusters compared to the original prototypes (diamonds).

Performance Metrics

The proposed method (NN + ST + Aux) achieved 21.2% accuracy on HMDB51 (vs. 13.0% for standard nearest-neighbor) and 18.6% on UCF101, outperforming the classic Attribute-based (DAP) method's 14.3%.

Conclusion and Future Work

While the achieved accuracies represent a significant leap for zero-shot learning on complex datasets, they remain below supervised performance levels. The reliance on low-level features like Dense Trajectories suggests that integrating end-to-end deep learning features could further improve results. Additionally, future research may focus on fine-tuning word embeddings specifically for the visual domain to resolve semantic ambiguities present in general-purpose text corpora.

Summary updated

BibTeX
@inproceedings{xu2015semantic,
  author = {Xun Xu and Timothy M. Hospedales and Shaogang Gong},
  title = {{Semantic embedding space for zero-shot action recognition}},
  booktitle = {ICIP},
  year = {2015},
  doi = {10.1109/icip.2015.7350760},
}
Figure