TL;DR
This paper introduces a novel approach to Zero-Shot Action Recognition (ZSAR) by translating both videos and class labels into natural language descriptions. By using video captioning models to describe video content and internet search engines to describe action classes, the method projects both modalities into a shared semantic space using a pre-trained BERT paraphrase embedder, achieving state-of-the-art performance without requiring any training on the target action classes.
What Problem Does This Paper Address?
Traditional Zero-Shot Action Recognition (ZSAR) methods attempt to map complex spatio-temporal video features directly to simple word vectors (like Word2Vec or GloVe) representing class labels. This creates a severe "domain adaptation problem" or semantic gap. For instance, completely different actions like "horse riding" and "pommel horse" might end up close to each other in a word vector space simply because they share the word "horse," despite being visually distinct.
The authors tackle this challenge by moving away from raw deep features and word vectors, proposing instead to represent both videos and labels using the same rich modality: natural language sentences.
Methods
The core innovation is a pipeline that unifies visual and semantic data into natural language descriptions before performing classification:
- Video Representation (Observers): Instead of using raw visual features, the video is passed through multiple "observers"—state-of-the-art video captioning architectures (Transformer and Bi-Modal Transformer) pre-trained on the ActivityNet Captions dataset. These generate descriptive sentences detailing the objects, scenes, and human actions in the video.
- Class Label Representation: Class labels are expanded into descriptive paragraphs by scraping documents from the internet via search engines. These paragraphs are split into sentences and filtered using Sentence-BERT (SBERT) to keep only the most contextually relevant descriptions.
- Joint Embedding Space: Both the video captions and the class descriptions are projected into a shared semantic space using a pre-trained BERT-based paraphrase embedder (
paraphrase-distilroberta-base-v2). Because both inputs are now sentences, determining the action class becomes a straightforward text similarity problem solved using the Nearest Neighbor rule with cosine similarity.
This framework is entirely cross-dataset and "plug-and-play," meaning captioning models or text encoders can be swapped out without retraining the downstream classifier.
Datasets
- ActivityNet Captions: Used exclusively to pre-train the video captioning models (observers) to generate fluent descriptions without using any action labels from the target datasets.
- UCF101 & HMDB51: Used for testing the zero-shot action recognition performance.
- TruZe Protocol: The authors adopt the strict TruZe evaluation protocol, which carefully removes any overlapping classes between the pre-training dataset (Kinetics400, which influences many standard feature extractors) and the test sets to ensure a true zero-shot scenario. This results in a 67/34 split for UCF101 and a 29/22 split for HMDB51.
Key Insights and Ideas
- Sentences Beat Word Vectors: Moving from word vectors (like GloVe) to full descriptive sentences significantly mitigates the domain adaptation problem. Sentences preserve contextual relationships between humans, objects, and scenes that single-word embeddings destroy.
- The Power of Multiple Observers: Combining multiple captioning models (observers) acting on different input modalities (RGB, optical flow, audio, and visual global vectors) yields complementary descriptions, boosting final classification accuracy significantly compared to any single observer.
- Paraphrase Models are Key: Standard sentence encoders like Sent2Vec fail dramatically in this setup because they cannot bridge the gap between video descriptions and class definitions. Pre-trained paraphrase models (like SBERT) excel because they are explicitly optimized to recognize when two different sentences describe the same underlying concept.
- Video Captioning is the Bottleneck: The authors show a strong correlation ($r > 0.9$) between video captioning metrics (like METEOR) and final ZSAR accuracy. This means the method's performance will naturally scale up as the computer vision community develops better video captioning models.
Key Results
- Under the strict TruZe protocol, the proposed method achieves a state-of-the-art accuracy of 49.1% on the UCF101 dataset, outperforming the previous best method (CLASTER at 45.8%) by 3.3 percentage points.
- On the HMDB51 dataset, it achieves a competitive accuracy of 20.4%, proving robust even though HMDB51 features very short clips that make video captioning highly challenging.
- In conventional 0%/50% cross-dataset splits, the method achieves 49.0% on UCF101 and 28.3% on HMDB51, outperforming several established object- and attribute-based baselines.
Limitations and Future Work
- Short Video Performance: The method struggles with very short video clips (like those in HMDB51, averaging 3.2 seconds), as short sequences provide fewer feature stacks, which can nullify the benefits of self-attention mechanisms in captioning models.
- Inter-Class Similarity: Highly similar action classes (e.g., "boxing punching bag" vs. "boxing speed bag", or face-related actions like "applying eye makeup" vs. "applying lipstick") frequently cause confusion. The text descriptions generated by the observers are sometimes not granular enough to distinguish these subtle visual differences.
- Future Directions: The authors suggest that developing end-to-end models that jointly optimize both video captioning and zero-shot action recognition objectives could significantly improve performance, alongside incorporating explicit object-relationship mapping to handle fine-grained distinctions.
Summary updated