Abstract
Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to a given question. Our proposed approach, FrozenBiLM, outperforms the state of the art in zero-shot VideoQA by a significant margin on a variety of datasets.
Introduction
Video Question Answering (VideoQA) represents a significant challenge in artificial intelligence, necessitating the integration of temporal visual cues, speech, and linguistic context. Traditional supervised approaches rely on large-scale, manually annotated datasets of video-question-answer triplets. However, the high cost and poor scalability of manual annotation have led researchers toward zero-shot learning.
Recent efforts in zero-shot VideoQA have focused on adapting "frozen" autoregressive language models (such as GPT-3) to multi-modal inputs. While effective, these models are computationally intensive and primarily optimized for next-token prediction. This paper introduces FrozenBiLM, a framework that leverages bidirectional language models (BiLMs) to provide a more efficient and accurate alternative for zero-shot VideoQA.
Methods
The Architecture
FrozenBiLM utilizes a pretrained bidirectional language model, specifically DeBERTa-V2-XLarge (900M parameters), which remains frozen to preserve its extensive linguistic knowledge. Visual features are extracted from video frames using a frozen CLIP ViT-L/14 encoder.
To bridge the gap between visual and textual modalities, the authors introduce lightweight trainable modules:
- Visual-Text Projection: A linear layer that maps visual features into the language embedding space.
- Adapters: Small bottleneck MLP modules inserted within the transformer blocks.
Remarkably, only approximately 5% of the total parameters are trained, ensuring computational efficiency and preventing the "catastrophic forgetting" of the language model's original capabilities.
Cloze-style Inference
A key innovation of FrozenBiLM is treating VideoQA as a cloze test (fill-in-the-blank). Instead of generating a sequence, the model converts a question into a prompt with a [MASK] token. For example, the question "What is the dog doing?" becomes "Question: What is the dog doing? Answer: [MASK]." The model then predicts the most likely token to fill the mask based on the combined visual and textual context.
Training Strategy
The model is trained on WebVid10M, a dataset comprising 10 million web-scraped video-caption pairs. During training, the model learns to predict masked words in captions while observing the corresponding video. Crucially, no manually annotated QA pairs are used during this phase, maintaining the zero-shot nature of the eventual QA task.
Key Insights and Results
The Advantage of Bidirectional Modeling
The core finding is that bidirectional models are inherently better suited for reasoning tasks than autoregressive ones. By considering context from both sides of the [MASK] token, BiLMs can more effectively anchor answers to the specific constraints of a question.
As shown in the ablation study, the combination of a pretrained, frozen LM with adapters consistently yields the best performance across diverse datasets. Freezing the LM is particularly critical for maintaining performance on complex benchmarks like iVQA and LSMDC.
Performance Benchmarks
FrozenBiLM establishes new state-of-the-art (SOTA) records for zero-shot VideoQA:
- MSVD-QA: Accuracy improved from 13.5% (previous SOTA) to 33.8%.
- iVQA: Achieved 26.8% accuracy, more than doubling previous zero-shot results.
- Efficiency: FrozenBiLM (900M parameters) outperformed much larger models like GPT-J (6B parameters) while requiring significantly fewer GPU hours for training (160 vs. 820).
Limitations and Future Work
While FrozenBiLM excels at concise QA, its masked modeling nature makes it unsuitable for long-form generation tasks like video captioning. Additionally, while it handles object and action identification well, complex temporal reasoning remains a challenge. Future work may involve scaling to larger BiLMs to explore potential emergent reasoning capabilities.
Summary updated