Is a Video worth $n\times n$ Images? A Highly Efficient Approach to Transformer-based Video Question Answering

Chenyang Lyu, Tianbo Ji, Yvette Graham, Jennifer Foster

TL;DR

By concatenating $n^2$ video frames into a single $n \times n$ grid image, this paper introduces a highly efficient "Single-Frame" approach to Video Question Answering (VideoQA). This simple yet effective trick allows standard image encoders to process an entire video sequence in a single forward pass, achieving state-of-the-art accuracy with a 4x speedup and a 70% reduction in memory usage.

What Problem Does This Paper Address?

Video Question Answering (VideoQA) is computationally expensive. Traditional Transformer-based models typically handle video in one of two ways:

  1. Independent Encoding: They run each frame through an image encoder separately. If you have 16 frames, you perform 16 forward passes. This is slow.
  2. Joint Encoding: They flatten all patches from all frames into one long sequence. Because Transformer attention complexity grows quadratically with sequence length, this approach quickly hits memory limits.

These bottlenecks make VideoQA research difficult for those without massive compute budgets and slow down real-world inference. The authors set out to find a way to maintain the temporal information of a video while drastically reducing the number of encoder calls.

Methods

The core innovation is the Frames Transformation strategy. Instead of treating a video as a sequence of separate images, the authors treat it as a single "mega-image."

  • Grid Concatenation: They sample $n^2$ frames from a video and arrange them into an $n \times n$ matrix (e.g., a $3 \times 3$ grid for 9 frames).
  • Single Forward Pass: This grid is converted into one image and fed into a pre-trained Vision Transformer (ViT) from CLIP. This reduces the image encoder usage from $n^2$ calls to just 1.
  • Architecture: The model uses a standard CLIP-based setup: a ViT for the video grid and a textual transformer for the question. The resulting video representation ($V^h$) and question representation ($Q^h$) interact via a Multi-Head Attention layer to produce the final answer.
Comparison between conventional multi-frame encoding and the proposed single-frame grid approach.

Datasets

The approach was evaluated on two major multi-choice VideoQA benchmarks:

  • MSRVTT-MC: A dataset where each video is paired with a question and 5 candidate options.
  • TrafficQA: A dataset focused on reasoning over traffic events, providing 4 options per question.

Key Insights and Ideas

  • Spatial-Temporal Trade-off: By arranging frames in a grid, the model's self-attention mechanism can naturally capture temporal relationships by looking at different "cells" of the grid, effectively treating time as just another spatial dimension.
  • Arrangement Matters: The authors tested different ways to stitch frames together. They found that a Matrix (Grid) layout significantly outperforms simple horizontal or vertical stacking. Stacking frames in a long strip (e.g., $1 \times 9$) distorts the visual information, whereas a $3 \times 3$ matrix preserves the aspect ratio and features of individual frames better.
  • Constant Computational Cost: Perhaps the most striking finding is shown in their efficiency analysis. As the number of frames increases, the computational time for the "Single-Frame" approach remains almost perfectly flat, while the traditional "Multi-Frame" approach scales linearly.
Figure 3: Evaluation results including accuracy (↑) and computational time (↓) on the effect of amount of video frames on MSRVTT-MC.

Key Results

The "Single-Frame" approach proved that you don't need massive compute to reach the top of the leaderboard:

  • Accuracy: On MSRVTT-MC, the model achieved 92.2% accuracy, slightly surpassing the previous state-of-the-art (92.1%) held by models like VideoCLIP and Singularity.
  • Speed: The training and inference speed was approximately 3.8x to 3.9x faster than the multi-frame baseline.
  • Efficiency: The model required only 30% of the memory compared to traditional methods, making it much more accessible for researchers with limited hardware (like a single NVIDIA GTX 3090).

Limitations and Future Work

While the results are impressive, the authors acknowledge a few hurdles:

  • Video Length: The datasets used (MSRVTT and TrafficQA) consist of relatively short clips. It remains to be seen how well a grid-based approach scales to very long videos where $n^2$ might need to be very large, potentially leading to extremely high-resolution "mega-images" or loss of detail.
  • Task Variety: The study focused strictly on VideoQA. Future work is needed to see if this "video-as-a-grid" approach holds up for other tasks like video retrieval or action recognition.
  • Resolution Limits: There is a physical limit to how many frames you can cram into one image before the resolution of each individual frame becomes too low for the encoder to extract meaningful features.

Summary updated

Abstract

Conventional Transformer-based Video Question Answering (VideoQA) approaches generally encode frames independently through one or more image encoders followed by interaction between frames and question. However, such schema would incur significant memory use and inevitably slow down the training and inference speed. In this work, we present a highly efficient approach for VideoQA based on existing vision-language pre-trained models where we concatenate video frames to a $n\times n$ matrix and then convert it to one image. By doing so, we reduce the use of the image encoder from $n^{2}$ to $1$ while maintaining the temporal structure of the original video. Experimental results on MSRVTT and TrafficQA show that our proposed approach achieves state-of-the-art performance with nearly $4\times$ faster speed and only 30% memory use. We show that by integrating our approach into VideoQA systems we can achieve comparable, even superior, performance with a significant speed up for training and inference. We believe the proposed approach can facilitate VideoQA-related research by reducing the computational requirements for those who have limited access to budgets and resources. Our code will be made publicly available for research use.
BibTeX
@inproceedings{lyu2023is,
  author = {Chenyang Lyu and Tianbo Ji and Yvette Graham and Jennifer Foster},
  title = {{Is a Video worth $n\textbackslash\{\}times n$ Images? A Highly Efficient Approach to Transformer-based Video Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure