Orthogonal Temporal Interpolation for Zero-Shot Video Recognition

Shuhui Wang(王树徽), Yan Zhu, Junbao Zhuo, Bin Ma, Jiajia Geng, Xiaoming Wei, Xiaolin Wei, Shuhui Wang

TL;DR

This paper tackles the challenge of Zero-Shot Video Recognition (ZSVR) by proposing a novel "Orthogonal Temporal Interpolation" (OTI) module. OTI addresses a surprising issue where standard temporal modeling in vision-language models actually hurts performance on unseen video categories. By carefully factoring out and interpolating orthogonal temporal features, OTI significantly improves recognition accuracy, achieving state-of-the-art results.

What Problem Does This Paper Address?

Zero-Shot Video Recognition (ZSVR) aims to classify videos into categories that the model has never seen during training. This is a crucial capability for real-world applications where new categories emerge constantly. Vision-Language Models (VLMs) like CLIP have shown great promise for ZSVR by aligning visual and textual information. However, applying these to videos requires incorporating temporal information. Existing methods typically add a temporal learning module (e.g., a transformer) after the initial visual encoding. The authors discovered a peculiar problem: for unseen categories, models using these temporal modules performed worse than models that relied solely on spatial features. They hypothesized that improper temporal modeling was actually disrupting the valuable spatial information. This paper seeks to understand and resolve this issue to enable more effective ZSVR.

Methods

The core of the proposed solution lies in the Orthogonal Temporal Interpolation (OTI) module. The authors' approach involves:

  1. Feature Factorization: They first decompose the spatial-temporal feature (obtained from a temporal module) into its spatial component and a residual temporal component. Crucially, they ensure this temporal component is orthogonal to the spatial feature. This is achieved by projecting the spatial-temporal feature onto the spatial feature and then subtracting it to get the orthogonal temporal feature.
  2. Orthogonal Temporal Interpolation: The orthogonal temporal feature is then interpolated with the original spatial feature using a learned weight. This creates a refined spatial-temporal feature. The intuition is that by isolating and carefully re-integrating the temporal information, they can avoid disrupting the spatial cues.
  3. Matching Loss: To further improve the quality of the orthogonal temporal feature, a "Matching Loss" is introduced. This loss encourages the temporal feature to be closer to the original spatial feature, preventing it from becoming too noisy or irrelevant.
  4. Overall Framework: The OTI module is integrated into a VLM-based framework (specifically using CLIP as the backbone). The model is trained with a combination of cross-entropy loss (for classification) and the proposed matching loss.

Datasets

The paper uses the following datasets:

  • Training: Kinetics-400 (used as seen categories).
  • Evaluation:
    • UCF101
    • HMDB51
    • Kinetics-600 (evaluated on new categories not present in Kinetics-400).

Key Insights and Ideas

  • The "Abnormal Phenomenon": The most striking insight is the discovery that adding standard temporal modeling to VLMs can degrade ZSVR performance on unseen categories. This challenges the common assumption that more temporal information is always better.
  • Temporal Disruption: The authors' hypothesis that improper temporal modeling disrupts spatial features is a key takeaway. This suggests a need for more nuanced temporal integration rather than simply adding a temporal module.
  • Orthogonality as a Solution: The concept of factoring out an orthogonal temporal feature is novel. This allows for a cleaner separation of temporal and spatial information, preventing interference.
  • Interpolation for Refinement: Carefully interpolating the orthogonal temporal feature back with the spatial feature, rather than just concatenating or adding them, proves effective in creating a more robust representation.
  • Matching Loss for Temporal Quality: The matching loss is crucial for ensuring the learned temporal features are meaningful and don't just add noise.

Key Results

  • State-of-the-Art Performance: The proposed OTI model achieves state-of-the-art results on UCF101, HMDB51, and Kinetics-600 datasets across various evaluation protocols.
  • Significant Improvements: For example, with the VIT-L/14 backbone, OTI achieves 92.8% accuracy on UCF101 (EP2) and 64.0% on HMDB51 (EP2), outperforming previous methods by a considerable margin.
  • Effectiveness of OTI Components: Ablation studies demonstrate that the combination of the orthogonal temporal interpolation and the matching loss is essential for achieving these gains. The model trained with all components (Lcls + Loti + Lmatch) significantly outperforms variants trained with only Lcls or Lcls + Loti.
  • Temporal Module Depth: Increasing the number of layers in the temporal learning module did not improve ZSVR performance, reinforcing the idea that the way temporal information is handled is more important than its complexity.

Limitations and Future Work

  • Temporal Module Depth: The paper notes that increasing the temporal learning module from one to two layers did not yield improvements, suggesting that deeper temporal modules might not be beneficial for this specific ZSVR task. This could be an area for future exploration.
  • Interpolation Weight: While the authors find optimal interpolation weights (λ) for different benchmarks, the exact mechanism for determining this optimal weight could be further investigated.
  • Generalizability: While the model shows strong performance on standard datasets, its performance on more diverse or challenging video domains could be explored in future work. The authors suggest that their approach is stable across different sampling methods, which is a positive sign for generalizability.

Summary updated

Abstract

Zero-shot video recognition (ZSVR) is a task that aims to recognize video categories that have not been seen during the model training process. Recently, vision-language models (VLMs) pre-trained on large-scale image-text pairs have demonstrated impressive transferability for ZSVR. To make VLMs applicable to the video domain, existing methods often use an additional temporal learning module after the image-level encoder to learn the temporal relationships among video frames. Unfortunately, for video from unseen categories, we observe an abnormal phenomenon where the model that uses spatial-temporal feature performs much worse than the model that removes temporal learning module and uses only spatial feature. We conjecture that improper temporal modeling on video disrupts the spatial feature of the video. To verify our hypothesis, we propose Feature Factorization to retain the orthogonal temporal feature of the video and use interpolation to construct refined spatial-temporal feature. The model using appropriately refined spatial-temporal feature performs better than the one using only spatial feature, which verifies the effectiveness of the orthogonal temporal feature for the ZSVR task. Therefore, an Orthogonal Temporal Interpolation module is designed to learn a better refined spatial-temporal video feature during training. Additionally, a Matching Loss is introduced to improve the quality of the orthogonal temporal feature. We propose a model called OTI for ZSVR by employing orthogonal temporal interpolation and the matching loss based on VLMs. The ZSVR accuracies on popular video datasets (i.e., Kinetics-600, UCF101 and HMDB51) show that OTI outperforms the previous state-of-the-art method by a clear margin.Our codes are publicly available at https://github.com/yanzhu/mm2023_oti.
BibTeX
@inproceedings{wang2023orthogonal,
  author = {Shuhui Wang(王树徽) and Yan Zhu and Junbao Zhuo and Bin Ma and Jiajia Geng and Xiaoming Wei and Xiaolin Wei and Shuhui Wang},
  title = {{Orthogonal Temporal Interpolation for Zero-Shot Video Recognition}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611903},
}

Loading related content...

Figure