Alternative Semantic Representations for Zero-Shot Human Action Recognition

Qian Wang, Ke Chen

Abstract

A proper semantic representation for encoding side information is key to the success of zero-shot learning. In this paper, we explore two alternative semantic representations especially for zero-shot human action recognition: textual descriptions of human actions and deep features extracted from still images relevant to human actions. Such side information are accessible on Web with little cost, which paves a new way in gaining side information for large-scale zero-shot human action recognition. We investigate different encoding methods to generate semantic representations for human actions from such side information. Based on our zero-shot visual recognition method, we conducted experiments on UCF101 and HMDB51 to evaluate two proposed semantic representations. The results suggest that our proposed text- and image-based semantic representations outperform traditional attributes and word vectors considerably for zero-shot human action recognition. In particular, the image-based semantic representations yield the favourable performance even though the representation is extracted from a small number of images per class.

Introduction

Zero-Shot Learning (ZSL) is often considered the "holy grail" of computer vision, aiming to recognize categories like "playing the piano" without ever seeing a training video of that specific action. To achieve this, models rely on side information—a semantic bridge connecting visual features to conceptual categories.

Traditionally, researchers have used manual attributes or shallow word vectors (e.g., word2vec). However, manual attributes are labor-intensive and subjective, while word vectors often fail to capture the visual reality of an action, leading to a significant "semantic gap." This paper proposes leveraging richer, web-accessible side information: textual descriptions and still images.

Methodology

The researchers propose two alternative semantic representations sourced entirely from the web:

  1. Text-based Representations: Instead of simple labels, the model uses full instructional articles from sites like WikiHow. These are encoded using methods like the Fisher Word Vector (FWV), which uses a Gaussian Mixture Model to capture the distribution and nuance of the text.
  2. Image-based Representations: Still images related to actions are retrieved via search engines. Features extracted from a pre-trained GoogLeNet are encoded into a Fisher Feature Vector (FFV).
Examples of textual descriptions and still images used as side information for action categories.

These representations are integrated into the BiDiLEL (Bidirectional Latent Embedding Learning) framework, which maps both visual and semantic features into a shared latent space for comparison.

The BiDiLEL framework mapping visual and semantic spaces into a shared latent space.

Experimental Results

The approach was evaluated on the UCF101 and HMDB51 benchmarks. The results demonstrate a dramatic improvement over traditional methods:

  • Performance: On UCF101, the image-based FFV reached 40.12% accuracy, nearly doubling the performance of standard word vectors (19.42%) and beating manual attributes (21.54%).
  • Efficiency: High accuracy is achievable with just 5 to 20 images per class, showing that massive datasets are not required to bridge the semantic gap.
  • Generalization: In Generalized ZSL settings, the image-based representations maintained significantly higher performance than other methods.

Discussion and Insights

A surprising insight is that still images are incredibly effective at describing actions, despite lacking temporal data. A single photo provides enough visual context to outperform dictionary definitions. Furthermore, using Fisher Vectors to model the distribution of data proved superior to simple averaging, suggesting that the diversity of cues is vital.

While successful, the authors noted limitations in capturing textual nuance (e.g., ignoring word order) and challenges in effectively fusing multi-modal data. Future work may involve fine-tuning models specifically on action-related imagery to further close the semantic gap.

Summary updated

Abstract

A proper semantic representation for encoding side information is key to the success of zero-shot learning. In this paper, we explore two alternative semantic representations especially for zero-shot human action recognition: textual descriptions of human actions and deep features extracted from still images relevant to human actions. Such side information are accessible on Web with little cost, which paves a new way in gaining side information for large-scale zero-shot human action recognition. We investigate different encoding methods to generate semantic representations for human actions from such side information. Based on our zero-shot visual recognition method, we conducted experiments on UCF101 and HMDB51 to evaluate two proposed semantic representations . The results suggest that our proposed text- and image-based semantic representations outperform traditional attributes and word vectors considerably for zero-shot human action recognition. In particular, the image-based semantic representations yield the favourable performance even though the representation is extracted from a small number of images per class.
BibTeX
@inproceedings{wang2017alternative,
  author = {Qian Wang and Ke Chen},
  title = {{Alternative Semantic Representations for Zero-Shot Human Action Recognition}},
  booktitle = {Ecml/pkdd},
  year = {2017},
  doi = {10.1007/978-3-319-71249-9\_6},
}
Figure