ReGen: A good Generative zero-shot video classifier should be Rewarded

Adrian Bulat, Enrique Sanchez, Brais Martínez, Georgios Tzimiropoulos

Introduction

Open-world (zero-shot) recognition remains a significant challenge in computer vision, particularly for video action recognition. The goal is to identify actions in videos that were not present during the model's training phase. Current methods typically fall into two categories: discriminative models, which excel at classification but often overfit to base classes, and generative models (captioners), which offer flexibility but often produce descriptions too generic for precise classification.

The Challenge: Bridging the Gap

When fine-tuning a generative captioner to output specific class names, it frequently suffers from "base class overfitting." The model loses its descriptive flexibility and becomes a rigid, underperforming classifier. ReGen addresses this by transforming generative video captioning models into high-performance zero-shot classifiers using a reinforcement learning framework.

The ReGen Framework

ReGen utilizes the REINFORCE algorithm to "reward" a generative model (such as GIT or BLIP) for producing captions that are discriminative, video-specific, and grammatically correct.

The ReGen training framework illustrating the three-fold reward system: CLS-R, CLIP-R, and GRAMMAR-R.

The framework employs three distinct reward functions:

  1. Classification Reward (CLS-R): Uses a frozen CLIP text encoder to ensure the generated caption's embedding is closer to the ground-truth class name than to others.
  2. Video-Specific Reward (CLIP-R): Ensures the caption remains grounded in the visual features of the specific input video, preventing the model from simply memorizing class labels.
  3. Grammar Reward (GRAMMAR-R): Utilizes a pre-trained Language Model to maintain human-readability and grammatical integrity, compensating for the "bag-of-words" nature of CLIP.

Experimental Results

The models were trained on Kinetics-400 and evaluated on benchmarks including HMDB-51, UCF-101, and Kinetics-220. ReGen consistently outperformed existing generative state-of-the-art methods across different architectures.

Comparison of ReGen against the REST baseline across multiple datasets and architectures (BLIP and GIT).

As shown in the table above, ReGen achieves significant gains in zero-shot accuracy. For instance, on HMDB-51, ReGen (GIT) reached 55.1% accuracy, a substantial improvement over the REST baseline. Furthermore, in few-shot settings, ReGen improved performance on HMDB-51 by 8.6% over previous methods.

Key Insights

  • Holistic Evaluation: By evaluating the entire caption rather than word-by-word, the RL approach allows for greater exploration of sentence structures and vocabulary.
  • Increased Diversity: ReGen-trained models exhibit a 20% to 100% increase in unique word usage per class, discovering descriptive terms beyond the training set.
  • Visual Grounding: Saliency map analysis indicates that ReGen models focus more on actual video pixels rather than relying solely on previously generated text tokens.

Conclusion

ReGen demonstrates that rewarding generative models for specific desirable traits—discrimination, grounding, and grammar—can effectively bridge the gap between flexible captioning and precise zero-shot classification. While dependent on the quality of underlying CLIP embeddings, it sets a new benchmark for open-world action recognition.

Summary updated

Abstract

This paper sets out to solve the following problem: How can we turn a generative video captioning model into an open-world video/action classification model? Video captioning models can naturally produce open-ended free-form descriptions of a given video which, however, might not be discriminative enough for video/action recognition. Unfortunately, when fine-tuned to auto-regress the class names directly, video captioning models overfit the base classes losing their open-world zero-shot capabilities. To alleviate base class overfitting, in this work, we propose to use reinforcement learning to enforce the output of the video captioning model to be more class-level discriminative. Specifically, we propose ReGen, a novel reinforcement learning based framework with a three-fold objective and reward functions: (1) a class-level discrimination reward that enforces the generated caption to be correctly classified into the corresponding action class, (2) a CLIP reward that encourages the generated caption to continue to be descriptive of the input video (i.e. video-specific), and (3) a grammar reward that preserves the grammatical correctness of the caption. We show that ReGen can train a model to produce captions that are: discriminative, video-specific and grammatically correct. Importantly, when evaluated on standard benchmarks for zero- and few-shot action classification, ReGen significantly outperforms the previous state-of-the-art.
BibTeX
@inproceedings{bulat2023regen,
  author = {Adrian Bulat and Enrique Sanchez and Brais Martínez and Georgios Tzimiropoulos},
  title = {{ReGen: A good Generative zero-shot video classifier should be Rewarded}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01244},
}
Figure