TL;DR
This paper tackles the challenge of zero-shot human action recognition by proposing and evaluating novel semantic representations derived from textual descriptions and images, outperforming traditional attributes and word vectors. The key insight is that these richer, web-sourced representations can bridge the semantic gap more effectively, leading to significant improvements in recognizing actions unseen during training.
What Problem Does This Paper Address?
Zero-shot learning (ZSL) aims to recognize objects or actions that were not present in the training data. This is achieved by transferring knowledge from seen classes to unseen classes using auxiliary "side information" that describes the classes. For human action recognition, traditional side information like attributes (e.g., "has arms," "is running") and word vectors (e.g., word embeddings for "running") have limitations. Attributes are often subjective, labor-intensive to define for a large number of actions, and may not capture nuanced visual differences. Word vectors, while easier to obtain, suffer from a "semantic gap" – they don't fully capture the visual essence of an action and can be ambiguous. This paper addresses the need for more effective semantic representations that are easily obtainable and better capture the visual semantics of human actions, especially for large-scale ZSL.
Methods
The authors propose two main categories of alternative semantic representations:
- Text-based Semantic Representations:
- Term-Document Matrix (TD): A basic representation where each action is represented by a vector indicating the frequency of words in its textual description.
- Average Word Vector (AWV): This method enhances word vectors by averaging the pre-trained word embeddings of all words in an action's textual description. This leverages the compositional nature of word vectors.
- Fisher Word Vector (FWV): A more sophisticated approach that models the distribution of word vectors within a document using a Gaussian Mixture Model (GMM). The Fisher Vector then captures the gradient of the log-likelihood with respect to the GMM parameters, providing a richer representation of the textual semantics.
- Image-based Semantic Representations:
- Average Feature Vector (AFV): Similar to AWV, this method averages deep image features extracted from a set of relevant still images for an action.
- Fisher Feature Vector (FFV): Analogous to FWV, this method uses Fisher Vectors to encode a set of deep image features extracted from relevant images, capturing the distribution of these visual features.
The core idea is to leverage readily available web resources (text descriptions and images) and encode them using these methods to create more discriminative semantic representations.
Datasets
The paper evaluates their proposed methods on two standard human action recognition datasets:
- UCF101: Contains 13,320 video clips across 101 action categories.
- HMDB51: Contains 6,766 video clips across 51 human action categories.
The authors use standard experimental protocols, including 5 random splits of seen/unseen classes for both datasets. They evaluate both conventional ZSL (testing only on unseen classes) and generalized ZSL (testing on both seen and unseen classes).
Key Insights and Ideas
- Web-sourced data is powerful: Textual descriptions and relevant images from the internet are a rich and accessible source of side information for ZSL, overcoming the limitations of manual attribute annotation.
- Bridging the semantic gap: The proposed text and image-based representations are more effective at bridging the "semantic gap" between visual and semantic information compared to traditional word vectors.
- Image features are highly discriminative: Still images, despite lacking temporal information, provide abundant visual cues that, when encoded appropriately (e.g., with Fisher Vectors), can lead to superior performance in action recognition.
- Fewer images can suffice: Surprisingly, even a small number of relevant images (e.g., 5-20 per class) can yield significant performance improvements, making the approach practical.
- Fisher Vectors offer richer representations: For both text and image data, the Fisher Vector encoding method generally outperforms simpler averaging methods, suggesting that modeling the distribution of features is beneficial.
Key Results
- Text-based: AWV (Average Word Vector) performed best among the text-based methods, achieving accuracies of 24.38% on UCF101 and 21.80% on HMDB51 in the inductive conventional ZSL setting. This is a notable improvement over the baseline word vector (WV) accuracy of 19.42% and 15.26% respectively.
- Image-based: FFV (Fisher Feature Vector) with K=1 achieved the best performance among image-based methods, reaching 40.12% on UCF101 and 25.82% on HMDB51 (inductive conventional ZSL). This significantly outperformed AFV and traditional attributes.
- Comparison: In a direct comparison (Table 4), the proposed AWV and FFV consistently outperformed traditional attributes and word vectors in both conventional and generalized ZSL scenarios. FFV, in particular, showed the highest accuracy for zero-shot action recognition.
Limitations and Future Work
- Textual representation simplicity: The paper acknowledges that the Term-Document Matrix (TD) is a very simple encoding method, resulting in high dimensionality and sparsity. Future work could explore more advanced NLP techniques like Latent Dirichlet Allocation (LDA) or document2vec to improve text representation.
- Ignoring word order: The current text-based methods do not consider word order, which can be crucial for understanding sentence meaning. Future work could incorporate techniques that account for word order.
- CNN pre-training: Image features are extracted using CNNs pre-trained on ImageNet. The authors suggest that fine-tuning these models on specific human action image data could lead to even better performance.
- Combining representations: Preliminary experiments on combining text and image-based representations did not yield better results than using image-based representations alone. The authors believe further study into combination methods is warranted.
- Dataset limitations: For HMDB51, AWV showed only a slight improvement over WV, possibly due to the presence of actions that are difficult to describe with text.
Summary updated