TEST-V: TEst-time Support-set Tuning for Zero-shot Video Classification

Rui Yan, Jin Wang, Hongyu Qu, Xiaoyu Du, Dong Zhang, Jinhui Tang, Tieniu Tan

Abstract

Zero-shot video classification remains a significant challenge due to the modality gap between text labels and visual features. This paper introduces TEST-V, a training-free framework that dynamically constructs and refines a visual support set during inference. By leveraging Large Language Models (LLMs) for diverse action descriptions and text-to-video models for synthetic example generation, TEST-V "dilates" the reference library. It then "erodes" noise through learnable temporal weights tuned at test time. Experimental results across four major benchmarks demonstrate that TEST-V significantly outperforms existing state-of-the-art methods, effectively narrowing the semantic gap for unseen video categories.

Introduction

Zero-shot video classification aims to identify actions in videos without prior training on those specific classes. While Vision-Language Models (VLMs) like CLIP provide a strong foundation, they struggle with the "modality gap"—the inherent difference in representation between a concise text label (e.g., "diving") and a complex, high-dimensional video.

Existing approaches typically fall into two categories: Test-time Prompt Tuning (TPT), which optimizes text prompts to align better with the test video, and Support-set Alignment, which uses a fixed set of visual examples for comparison. However, TPT only addresses the text side of the gap, and static support sets often contain irrelevant or noisy samples that cannot adapt to specific test instances. TEST-V addresses these shortcomings by making the visual support set itself tunable during inference.

Comparison of different tuning strategies: (a) Text Prompt Tuning, (b) Support Set Alignment, and (c) the proposed Support Set Tuning (TEST-V).

Methodology

The TEST-V framework employs a "dilate and erode" strategy to bridge the modality gap through two primary modules: Multi-prompting Support-set Dilation (MSD) and Temporal-aware Support-set Erosion (TSE).

The overall architecture of TEST-V, illustrating the Multi-prompting Support-set Dilation (MSD) and Temporal-aware Support-set Erosion (TSE) modules.

Multi-prompting Support-set Dilation (MSD)

Instead of relying solely on a class name, MSD uses an LLM (e.g., ChatGPT) to generate diverse descriptions of an action. For instance, "golf" might be expanded into "a golfer lining up a shot" or "a golfer chipping onto the green." These descriptions are then processed by a text-to-video (T2V) model (e.g., LaVie) to generate a diverse set of synthetic video samples. This process "dilates" the search space, capturing various contexts and perspectives of the target action.

Temporal-aware Support-set Erosion (TSE)

Synthetic data is often imperfect. TSE "erodes" the support set by applying learnable weights at both the video and frame levels. Video-level weights determine which generated samples are most relevant to the test video, while frame-level weights identify the most critical moments within those samples. These weights are optimized during inference using a self-supervised loss based on "temporal prediction consistency," ensuring stable predictions across different temporal scales.

Experiments and Results

TEST-V was evaluated on four standard benchmarks: HMDB-51, UCF-101, Kinetics-600, and ActivityNet. The framework achieved state-of-the-art (SOTA) performance without any traditional training on labeled video data:

  • Performance Gains: TEST-V improved upon the baseline CLIP model by an average of 2.98% and surpassed other SOTA methods like BIKE and ViFi-CLIP by 2.15% and 1.83%, respectively.
  • Benchmark Scores: On HMDB-51, TEST-V reached 53.0% accuracy (with ViFi-CLIP), and 78.6% on UCF-101 (with BIKE).
  • Robustness: Visualization through t-SNE confirmed that MSD creates a more comprehensive feature space, allowing for better separation between similar classes compared to single-prompt methods.

Discussion and Insights

The success of TEST-V highlights the importance of diversity in visual references. By "brainstorming" scenarios via LLMs, the model builds a more robust boundary for classification. Furthermore, treating the support set as a learnable entity allows the model to effectively ignore "hallucinations" or noise in synthetic data that do not align with the test video's features. The use of temporal consistency as a self-supervised signal at test time proves to be a powerful mechanism for identifying pivotal visual cues.

Conclusion and Limitations

TEST-V presents a novel paradigm for zero-shot video classification by tuning visual support sets at test time. While effective, it faces challenges regarding computational cost, as spatio-temporal tuning increases inference time per video. Additionally, its performance is dependent on the quality of the underlying generative models (LLM and T2V). Future research will focus on optimizing inference efficiency and exploring lightweight generative models for real-time applications.

Summary updated

Abstract

Recently, adapting Vision Language Models (VLMs) to zero-shot visual classification by tuning class embedding with a few prompts (Test-time Prompt Tuning, TPT) or replacing class names with generated visual samples (support-set) has shown promising results. However, TPT cannot avoid the semantic gap between modalities while the support-set cannot be tuned. To this end, we draw on each other's strengths and propose a novel framework, namely TEst-time Support-set Tuning for zero-shot Video Classification (TEST-V). It first dilates the support-set with multiple prompts (Multi-prompting Support-set Dilation, MSD) and then erodes the support-set via learnable weights to mine key cues dynamically (Temporal-aware Support-set Erosion, TSE). Specifically, i) MSD expands the support samples for each class based on multiple prompts inquired from LLMs to enrich the diversity of the support-set. ii) TSE tunes the support-set with factorized learnable weights according to the temporal prediction consistency in a self-supervised manner to dig pivotal supporting cues for each class. TEST-V achieves state-of-the-art results across four benchmarks and shows good interpretability.
BibTeX
@inproceedings{yan2025test,
  author = {Rui Yan and Jin Wang and Hongyu Qu and Xiaoyu Du and Dong Zhang and Jinhui Tang and Tieniu Tan},
  title = {{TEST-V: TEst-time Support-set Tuning for Zero-shot Video Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/239},
}
Figure