Introduction
Open-world (zero-shot) recognition remains a significant challenge in computer vision, particularly for video action recognition. The goal is to identify actions in videos that were not present during the model's training phase. Current methods typically fall into two categories: discriminative models, which excel at classification but often overfit to base classes, and generative models (captioners), which offer flexibility but often produce descriptions too generic for precise classification.
The Challenge: Bridging the Gap
When fine-tuning a generative captioner to output specific class names, it frequently suffers from "base class overfitting." The model loses its descriptive flexibility and becomes a rigid, underperforming classifier. ReGen addresses this by transforming generative video captioning models into high-performance zero-shot classifiers using a reinforcement learning framework.
The ReGen Framework
ReGen utilizes the REINFORCE algorithm to "reward" a generative model (such as GIT or BLIP) for producing captions that are discriminative, video-specific, and grammatically correct.
The framework employs three distinct reward functions:
- Classification Reward (CLS-R): Uses a frozen CLIP text encoder to ensure the generated caption's embedding is closer to the ground-truth class name than to others.
- Video-Specific Reward (CLIP-R): Ensures the caption remains grounded in the visual features of the specific input video, preventing the model from simply memorizing class labels.
- Grammar Reward (GRAMMAR-R): Utilizes a pre-trained Language Model to maintain human-readability and grammatical integrity, compensating for the "bag-of-words" nature of CLIP.
Experimental Results
The models were trained on Kinetics-400 and evaluated on benchmarks including HMDB-51, UCF-101, and Kinetics-220. ReGen consistently outperformed existing generative state-of-the-art methods across different architectures.
As shown in the table above, ReGen achieves significant gains in zero-shot accuracy. For instance, on HMDB-51, ReGen (GIT) reached 55.1% accuracy, a substantial improvement over the REST baseline. Furthermore, in few-shot settings, ReGen improved performance on HMDB-51 by 8.6% over previous methods.
Key Insights
- Holistic Evaluation: By evaluating the entire caption rather than word-by-word, the RL approach allows for greater exploration of sentence structures and vocabulary.
- Increased Diversity: ReGen-trained models exhibit a 20% to 100% increase in unique word usage per class, discovering descriptive terms beyond the training set.
- Visual Grounding: Saliency map analysis indicates that ReGen models focus more on actual video pixels rather than relying solely on previously generated text tokens.
Conclusion
ReGen demonstrates that rewarding generative models for specific desirable traits—discrimination, grounding, and grammar—can effectively bridge the gap between flexible captioning and precise zero-shot classification. While dependent on the quality of underlying CLIP embeddings, it sets a new benchmark for open-world action recognition.
Summary updated