Video-Context Aligned Transformer for Video Question Answering

Linlin Zong, Jiahui Wan, Xianchao Zhang, Xinyue Liu, Wenxin Liang, Bo XuAAAI 2024 · DOI

Abstract

Video Question Answering (VideoQA) is a challenging multimodal task that requires a deep understanding of both visual and textual information. Current models often struggle with "information imbalance"—where a single question is too sparse to represent a dense video—and "semantic unalignment" between visual and textual features. We propose the Video-Context Aligned Transformer (V-CAT), a novel framework that utilizes all questions related to a video as a comprehensive textual context. By aligning this context with the video through a multi-stage transformer architecture and contrastive learning, V-CAT achieves state-of-the-art results on MSVD-QA and MSRVTT-QA benchmarks, demonstrating that architectural alignment can be more effective than massive data scaling.

Introduction

Video Question Answering (VideoQA) requires models to reason over temporal visual sequences to answer natural language queries. Despite significant progress, two major hurdles persist. First, Semantic Unalignment: visual features and textual features often reside in disparate embedding spaces, making it difficult to correlate specific actions, such as "pushing a car" (see Figure 1), with their visual counterparts. Second, Information Imbalance: a single question typically focuses on a tiny fraction of a video's content, leaving the model to search for a "needle in a haystack" amidst irrelevant background data.

Conceptual overview of semantic and content alignment.

To address these, V-CAT adopts a "reading comprehension" strategy. By treating the collection of all questions associated with a video as a "context," the model gains a richer, more balanced representation of the video's content before attempting to answer a specific query.

The V-CAT Framework

The V-CAT architecture is designed to progressively refine the relationship between video and text through four integrated modules.

The V-CAT architecture, illustrating the flow from feature extraction to answer decoding.

1. Feature Extractor

The model utilizes standard pre-trained backbones to generate initial embeddings. For video frames, it employs a combination of ResNet and ResNeXt to capture spatial and motion features. For textual input (both the specific question and the broader context), it uses BERT.

2. Semantic Alignment Module

This module serves as a global bridge. Using Contrastive Learning, it projects video and context features into a shared semantic space. By maximizing the similarity between matching video-context pairs and minimizing it for non-matching ones, the model learns a robust global understanding of the scene.

3. Content Alignment Module

Once global semantics are aligned, this module performs a "skimming" operation. It uses the global context to identify and aggregate relevant video features while filtering out redundant background information. This step ensures that the subsequent reasoning process focuses only on the most informative parts of the video.

4. Answer Decoder

The final stage involves a fine-grained interaction between the specific question and the refined, context-aligned video features. A transformer-based decoder processes these inputs to predict the final answer from a predefined vocabulary via a linear classifier.

Experimental Results

V-CAT was evaluated on two major open-ended VideoQA benchmarks: MSVD-QA and MSRVTT-QA. These datasets are particularly challenging as they require selecting answers from a large, open-ended vocabulary.

Performance Benchmarks

V-CAT achieved new state-of-the-art (SOTA) performance on both datasets:

  • MSVD-QA: Reached 45.2% accuracy, a 2.6% improvement over the previous best model (EIGV).
  • MSRVTT-QA: Reached 43.3% accuracy, outperforming high-profile models such as MERLOT (43.1%) and CoMVT (39.5%).

Notably, V-CAT outperforms models like HD-VILA and CoMVT which were pre-trained on massive datasets exceeding 100 million samples. This suggests that the context-alignment strategy is a highly efficient alternative to raw data scaling.

Ablation Study: The Power of Context

The core hypothesis of V-CAT—that context acts as an information equalizer—was confirmed through ablation studies. When the context module was removed, accuracy on MSVD-QA dropped from 45.2% to 35.7%. This 9.5% decrease underscores that the "noise" of additional questions actually provides critical signals for video understanding.

Conclusion

The Video-Context Aligned Transformer (V-CAT) demonstrates that leveraging auxiliary textual information (context) can significantly mitigate the information imbalance in VideoQA. By combining global semantic alignment with local content refinement, V-CAT provides a stable and efficient framework for multimodal reasoning. Future work may explore more sophisticated generative decoders or automated captioning to provide context in single-question scenarios.

Summary updated

BibTeX
@inproceedings{zong2024video,
  author = {Linlin Zong and Jiahui Wan and Xianchao Zhang and Xinyue Liu and Wenxin Liang and Bo Xu},
  title = {{Video-Context Aligned Transformer for Video Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29954},
}
Figure