Abstract
Recently, adapting Vision Language Models (VLMs) to zero-shot visual classification by tuning class embedding with a few prompts (Test-time Prompt Tuning, TPT) or replacing class names with generated visual samples (support-set) has shown promising results. However, TPT cannot avoid the semantic gap between modalities while the support-set cannot be tuned. To this end, we draw on each other's strengths and propose a novel framework namely TEst-time Support-set Tuning for zero-shot Video Classification (TEST-V). It first dilates the support-set with multiple prompts (Multi-prompting Support-set Dilation, MSD) and then erodes the support-set via learnable weights to mine key cues dynamically (Temporal-aware Support-set Erosion, TSE). Specifically, i) MSD expands the support samples for each class based on multiple prompts enquired from LLMs to enrich the diversity of the support-set. ii) TSE tunes the support-set with factorized learnable weights according to the temporal prediction consistency in a self-supervised manner to dig pivotal supporting cues for each class. TEST-V achieves state-of-the-art results across four benchmarks and has good interpretability for the support-set dilation and erosion.
Introduction
Zero-shot video classification—identifying actions in videos that a model has never seen during training—is a significant challenge in computer vision. While Vision-Language Models (VLMs) like CLIP have made massive strides, they still struggle with the "modality gap": the inherent difference between textual descriptions and visual temporal data.
Current approaches usually fall into two categories: Test-time Prompt Tuning (TPT), which adjusts text prompts to fit a test video, and Support-Set methods, which compare test videos to static synthetic samples. TPT fails to bridge the visual-textual gap, while static support sets cannot adapt to the specific nuances of a given test sample. TEST-V addresses these limitations by introducing a framework that is both diverse in its knowledge and adaptive at test-time.
Methodology
The TEST-V framework operates through a "dilation and erosion" strategy, consisting of two core phases designed to enrich and then refine the model's visual understanding.
Multi-prompting Support-set Dilation (MSD)
Before processing a test video, the model "dilates" its knowledge base. It leverages Large Language Models (LLMs) to generate multiple, detailed descriptions for every action class. For example, instead of a generic "golf" label, the LLM generates specific scenarios like "a golfer lines up their shot" or "a golfer chips the ball onto the green." These diverse prompts are then fed into a Text-to-Video (T2V) model to create a semantically rich and diverse "support set" of synthetic videos.
Temporal-aware Support-set Erosion (TSE)
At test-time, the model "erodes" the support set to identify the most relevant visual cues for the specific input video. This is achieved through factorized re-weighting, where the model learns weights for both synthetic videos and individual frames within those videos. Using a self-supervised tuning process, the model optimizes for "prediction consistency" across different temporal scales. This allows the model to effectively ignore irrelevant synthetic frames and amplify those that provide the best supporting evidence for the current test sample.
Experimental Results
TEST-V was evaluated on four major action recognition benchmarks: HMDB-51, UCF-101, Kinetics-600, and ActivityNet. The results demonstrate that TEST-V consistently outperforms existing state-of-the-art (SOTA) methods:
- Accuracy: On HMDB-51, TEST-V improved the baseline from 49.1% to 52.9%. On UCF-101, it achieved a high of 78.6%.
- Generalization: The method shows robust performance across different VLM backbones, such as ViT-B/16 and ViT-L/14.
- Efficiency: Despite the iterative test-time optimization, inference remains efficient, taking approximately 0.9 seconds per video on a standard GPU.
Conclusion
TEST-V successfully bridges the modality gap in zero-shot video classification by combining generative diversity with adaptive test-time tuning. By dilating the support set with LLM-guided variety and eroding it to find specific temporal cues, the framework achieves new SOTA results while maintaining interpretability. Future work will focus on further reducing computational costs for long-form video and improving the reliability of generative components.
Summary updated