Tell me what you see: A zero-shot action recognition method based on natural language descriptions

Valter Estevam, Rayson Laroca, David Menotti, Helio Pedrini

TL;DR

This paper introduces a novel approach to Zero-Shot Action Recognition (ZSAR) by translating both videos and class labels into natural language descriptions. By using video captioning models to describe video content and internet search engines to describe action classes, the method projects both modalities into a shared semantic space using a pre-trained BERT paraphrase embedder, achieving state-of-the-art performance without requiring any training on the target action classes.

What Problem Does This Paper Address?

Traditional Zero-Shot Action Recognition (ZSAR) methods attempt to map complex spatio-temporal video features directly to simple word vectors (like Word2Vec or GloVe) representing class labels. This creates a severe "domain adaptation problem" or semantic gap. For instance, completely different actions like "horse riding" and "pommel horse" might end up close to each other in a word vector space simply because they share the word "horse," despite being visually distinct.

The authors tackle this challenge by moving away from raw deep features and word vectors, proposing instead to represent both videos and labels using the same rich modality: natural language sentences.

Methods

The core innovation is a pipeline that unifies visual and semantic data into natural language descriptions before performing classification:

  1. Video Representation (Observers): Instead of using raw visual features, the video is passed through multiple "observers"—state-of-the-art video captioning architectures (Transformer and Bi-Modal Transformer) pre-trained on the ActivityNet Captions dataset. These generate descriptive sentences detailing the objects, scenes, and human actions in the video.
  2. Class Label Representation: Class labels are expanded into descriptive paragraphs by scraping documents from the internet via search engines. These paragraphs are split into sentences and filtered using Sentence-BERT (SBERT) to keep only the most contextually relevant descriptions.
  3. Joint Embedding Space: Both the video captions and the class descriptions are projected into a shared semantic space using a pre-trained BERT-based paraphrase embedder (paraphrase-distilroberta-base-v2). Because both inputs are now sentences, determining the action class becomes a straightforward text similarity problem solved using the Nearest Neighbor rule with cosine similarity.

This framework is entirely cross-dataset and "plug-and-play," meaning captioning models or text encoders can be swapped out without retraining the downstream classifier.

Overview of the proposed ZSAR method pipeline, showing how videos and class labels are converted to text and projected into a joint embedding space.

Datasets

  • ActivityNet Captions: Used exclusively to pre-train the video captioning models (observers) to generate fluent descriptions without using any action labels from the target datasets.
  • UCF101 & HMDB51: Used for testing the zero-shot action recognition performance.
  • TruZe Protocol: The authors adopt the strict TruZe evaluation protocol, which carefully removes any overlapping classes between the pre-training dataset (Kinetics400, which influences many standard feature extractors) and the test sets to ensure a true zero-shot scenario. This results in a 67/34 split for UCF101 and a 29/22 split for HMDB51.

Key Insights and Ideas

  • Sentences Beat Word Vectors: Moving from word vectors (like GloVe) to full descriptive sentences significantly mitigates the domain adaptation problem. Sentences preserve contextual relationships between humans, objects, and scenes that single-word embeddings destroy.
  • The Power of Multiple Observers: Combining multiple captioning models (observers) acting on different input modalities (RGB, optical flow, audio, and visual global vectors) yields complementary descriptions, boosting final classification accuracy significantly compared to any single observer.
  • Paraphrase Models are Key: Standard sentence encoders like Sent2Vec fail dramatically in this setup because they cannot bridge the gap between video descriptions and class definitions. Pre-trained paraphrase models (like SBERT) excel because they are explicitly optimized to recognize when two different sentences describe the same underlying concept.
  • Video Captioning is the Bottleneck: The authors show a strong correlation ($r > 0.9$) between video captioning metrics (like METEOR) and final ZSAR accuracy. This means the method's performance will naturally scale up as the computer vision community develops better video captioning models.

Key Results

  • Under the strict TruZe protocol, the proposed method achieves a state-of-the-art accuracy of 49.1% on the UCF101 dataset, outperforming the previous best method (CLASTER at 45.8%) by 3.3 percentage points.
  • On the HMDB51 dataset, it achieves a competitive accuracy of 20.4%, proving robust even though HMDB51 features very short clips that make video captioning highly challenging.
  • In conventional 0%/50% cross-dataset splits, the method achieves 49.0% on UCF101 and 28.3% on HMDB51, outperforming several established object- and attribute-based baselines.

Limitations and Future Work

  • Short Video Performance: The method struggles with very short video clips (like those in HMDB51, averaging 3.2 seconds), as short sequences provide fewer feature stacks, which can nullify the benefits of self-attention mechanisms in captioning models.
  • Inter-Class Similarity: Highly similar action classes (e.g., "boxing punching bag" vs. "boxing speed bag", or face-related actions like "applying eye makeup" vs. "applying lipstick") frequently cause confusion. The text descriptions generated by the observers are sometimes not granular enough to distinguish these subtle visual differences.
  • Future Directions: The authors suggest that developing end-to-end models that jointly optimize both video captioning and zero-shot action recognition objectives could significantly improve performance, alongside incorporating explicit object-relationship mapping to handle fine-grained distinctions.

Summary updated

Abstract

This paper presents a novel approach to Zero-Shot Action Recognition. Recent works have explored the detection and classification of objects to obtain semantic information from videos with remarkable performance. Inspired by them, we propose using video captioning methods to extract semantic information about objects, scenes, humans, and their relationships. To the best of our knowledge, this is the first work to represent both videos and labels with descriptive sentences. More specifically, we represent videos using sentences generated via video captioning methods and classes using sentences extracted from documents acquired through search engines on the Internet. Using these representations, we build a shared semantic space employing BERT-based embedders pre-trained in the paraphrasing task on multiple text datasets. The projection of both visual and semantic information onto this space is straightforward, as they are sentences, enabling classification using the nearest neighbor rule. We demonstrate that representing videos and labels with sentences alleviates the domain adaptation problem. Additionally, we show that word vectors are unsuitable for building the semantic embedding space of our descriptions. Our method outperforms the state-of-the-art performance on the UCF101 dataset by 3.3 p.p. in accuracy under the TruZe protocol and achieves competitive results on both the UCF101 and HMDB51 datasets under the conventional protocol (0/50\% - training/testing split). Our code is available at https://github.com/valterlej/zsarcap.
BibTeX
@article{estevam2021tell,
  author = {Valter Estevam and Rayson Laroca and David Menotti and Helio Pedrini},
  title = {{Tell me what you see: A zero-shot action recognition method based on natural language descriptions}},
  journal = {Multimedia Tools and Applications},
  year = {2021},
  doi = {10.1007/s11042-023-16566-5},
}
Figure