Abstract
Video Question Answering (VideoQA) is a challenging multimodal task that requires a deep understanding of both visual and textual information. Current models often struggle with "information imbalance"—where a single question is too sparse to represent a dense video—and "semantic unalignment" between visual and textual features. We propose the Video-Context Aligned Transformer (V-CAT), a novel framework that utilizes all questions related to a video as a comprehensive textual context. By aligning this context with the video through a multi-stage transformer architecture and contrastive learning, V-CAT achieves state-of-the-art results on MSVD-QA and MSRVTT-QA benchmarks, demonstrating that architectural alignment can be more effective than massive data scaling.
Introduction
Video Question Answering (VideoQA) requires models to reason over temporal visual sequences to answer natural language queries. Despite significant progress, two major hurdles persist. First, Semantic Unalignment: visual features and textual features often reside in disparate embedding spaces, making it difficult to correlate specific actions, such as "pushing a car" (see Figure 1), with their visual counterparts. Second, Information Imbalance: a single question typically focuses on a tiny fraction of a video's content, leaving the model to search for a "needle in a haystack" amidst irrelevant background data.
To address these, V-CAT adopts a "reading comprehension" strategy. By treating the collection of all questions associated with a video as a "context," the model gains a richer, more balanced representation of the video's content before attempting to answer a specific query.
The V-CAT Framework
The V-CAT architecture is designed to progressively refine the relationship between video and text through four integrated modules.
1. Feature Extractor
The model utilizes standard pre-trained backbones to generate initial embeddings. For video frames, it employs a combination of ResNet and ResNeXt to capture spatial and motion features. For textual input (both the specific question and the broader context), it uses BERT.
2. Semantic Alignment Module
This module serves as a global bridge. Using Contrastive Learning, it projects video and context features into a shared semantic space. By maximizing the similarity between matching video-context pairs and minimizing it for non-matching ones, the model learns a robust global understanding of the scene.
3. Content Alignment Module
Once global semantics are aligned, this module performs a "skimming" operation. It uses the global context to identify and aggregate relevant video features while filtering out redundant background information. This step ensures that the subsequent reasoning process focuses only on the most informative parts of the video.
4. Answer Decoder
The final stage involves a fine-grained interaction between the specific question and the refined, context-aligned video features. A transformer-based decoder processes these inputs to predict the final answer from a predefined vocabulary via a linear classifier.
Experimental Results
V-CAT was evaluated on two major open-ended VideoQA benchmarks: MSVD-QA and MSRVTT-QA. These datasets are particularly challenging as they require selecting answers from a large, open-ended vocabulary.
Performance Benchmarks
V-CAT achieved new state-of-the-art (SOTA) performance on both datasets:
- MSVD-QA: Reached 45.2% accuracy, a 2.6% improvement over the previous best model (EIGV).
- MSRVTT-QA: Reached 43.3% accuracy, outperforming high-profile models such as MERLOT (43.1%) and CoMVT (39.5%).
Notably, V-CAT outperforms models like HD-VILA and CoMVT which were pre-trained on massive datasets exceeding 100 million samples. This suggests that the context-alignment strategy is a highly efficient alternative to raw data scaling.
Ablation Study: The Power of Context
The core hypothesis of V-CAT—that context acts as an information equalizer—was confirmed through ablation studies. When the context module was removed, accuracy on MSVD-QA dropped from 45.2% to 35.7%. This 9.5% decrease underscores that the "noise" of additional questions actually provides critical signals for video understanding.
Conclusion
The Video-Context Aligned Transformer (V-CAT) demonstrates that leveraging auxiliary textual information (context) can significantly mitigate the information imbalance in VideoQA. By combining global semantic alignment with local content refinement, V-CAT provides a stable and efficient framework for multimodal reasoning. Future work may explore more sophisticated generative decoders or automated captioning to provide context in single-question scenarios.
Summary updated