Abstract
Zero-shot video classification remains a significant challenge due to the modality gap between text labels and visual features. This paper introduces TEST-V, a training-free framework that dynamically constructs and refines a visual support set during inference. By leveraging Large Language Models (LLMs) for diverse action descriptions and text-to-video models for synthetic example generation, TEST-V "dilates" the reference library. It then "erodes" noise through learnable temporal weights tuned at test time. Experimental results across four major benchmarks demonstrate that TEST-V significantly outperforms existing state-of-the-art methods, effectively narrowing the semantic gap for unseen video categories.
Introduction
Zero-shot video classification aims to identify actions in videos without prior training on those specific classes. While Vision-Language Models (VLMs) like CLIP provide a strong foundation, they struggle with the "modality gap"—the inherent difference in representation between a concise text label (e.g., "diving") and a complex, high-dimensional video.
Existing approaches typically fall into two categories: Test-time Prompt Tuning (TPT), which optimizes text prompts to align better with the test video, and Support-set Alignment, which uses a fixed set of visual examples for comparison. However, TPT only addresses the text side of the gap, and static support sets often contain irrelevant or noisy samples that cannot adapt to specific test instances. TEST-V addresses these shortcomings by making the visual support set itself tunable during inference.
Methodology
The TEST-V framework employs a "dilate and erode" strategy to bridge the modality gap through two primary modules: Multi-prompting Support-set Dilation (MSD) and Temporal-aware Support-set Erosion (TSE).
Multi-prompting Support-set Dilation (MSD)
Instead of relying solely on a class name, MSD uses an LLM (e.g., ChatGPT) to generate diverse descriptions of an action. For instance, "golf" might be expanded into "a golfer lining up a shot" or "a golfer chipping onto the green." These descriptions are then processed by a text-to-video (T2V) model (e.g., LaVie) to generate a diverse set of synthetic video samples. This process "dilates" the search space, capturing various contexts and perspectives of the target action.
Temporal-aware Support-set Erosion (TSE)
Synthetic data is often imperfect. TSE "erodes" the support set by applying learnable weights at both the video and frame levels. Video-level weights determine which generated samples are most relevant to the test video, while frame-level weights identify the most critical moments within those samples. These weights are optimized during inference using a self-supervised loss based on "temporal prediction consistency," ensuring stable predictions across different temporal scales.
Experiments and Results
TEST-V was evaluated on four standard benchmarks: HMDB-51, UCF-101, Kinetics-600, and ActivityNet. The framework achieved state-of-the-art (SOTA) performance without any traditional training on labeled video data:
- Performance Gains: TEST-V improved upon the baseline CLIP model by an average of 2.98% and surpassed other SOTA methods like BIKE and ViFi-CLIP by 2.15% and 1.83%, respectively.
- Benchmark Scores: On HMDB-51, TEST-V reached 53.0% accuracy (with ViFi-CLIP), and 78.6% on UCF-101 (with BIKE).
- Robustness: Visualization through t-SNE confirmed that MSD creates a more comprehensive feature space, allowing for better separation between similar classes compared to single-prompt methods.
Discussion and Insights
The success of TEST-V highlights the importance of diversity in visual references. By "brainstorming" scenarios via LLMs, the model builds a more robust boundary for classification. Furthermore, treating the support set as a learnable entity allows the model to effectively ignore "hallucinations" or noise in synthetic data that do not align with the test video's features. The use of temporal consistency as a self-supervised signal at test time proves to be a powerful mechanism for identifying pivotal visual cues.
Conclusion and Limitations
TEST-V presents a novel paradigm for zero-shot video classification by tuning visual support sets at test time. While effective, it faces challenges regarding computational cost, as spatio-temporal tuning increases inference time per video. Additionally, its performance is dependent on the quality of the underlying generative models (LLM and T2V). Future research will focus on optimizing inference efficiency and exploring lightweight generative models for real-time applications.
Summary updated