TL;DR
By concatenating $n^2$ video frames into a single $n \times n$ grid image, this paper introduces a highly efficient "Single-Frame" approach to Video Question Answering (VideoQA). This simple yet effective trick allows standard image encoders to process an entire video sequence in a single forward pass, achieving state-of-the-art accuracy with a 4x speedup and a 70% reduction in memory usage.
What Problem Does This Paper Address?
Video Question Answering (VideoQA) is computationally expensive. Traditional Transformer-based models typically handle video in one of two ways:
- Independent Encoding: They run each frame through an image encoder separately. If you have 16 frames, you perform 16 forward passes. This is slow.
- Joint Encoding: They flatten all patches from all frames into one long sequence. Because Transformer attention complexity grows quadratically with sequence length, this approach quickly hits memory limits.
These bottlenecks make VideoQA research difficult for those without massive compute budgets and slow down real-world inference. The authors set out to find a way to maintain the temporal information of a video while drastically reducing the number of encoder calls.
Methods
The core innovation is the Frames Transformation strategy. Instead of treating a video as a sequence of separate images, the authors treat it as a single "mega-image."
- Grid Concatenation: They sample $n^2$ frames from a video and arrange them into an $n \times n$ matrix (e.g., a $3 \times 3$ grid for 9 frames).
- Single Forward Pass: This grid is converted into one image and fed into a pre-trained Vision Transformer (ViT) from CLIP. This reduces the image encoder usage from $n^2$ calls to just 1.
- Architecture: The model uses a standard CLIP-based setup: a ViT for the video grid and a textual transformer for the question. The resulting video representation ($V^h$) and question representation ($Q^h$) interact via a Multi-Head Attention layer to produce the final answer.
Datasets
The approach was evaluated on two major multi-choice VideoQA benchmarks:
- MSRVTT-MC: A dataset where each video is paired with a question and 5 candidate options.
- TrafficQA: A dataset focused on reasoning over traffic events, providing 4 options per question.
Key Insights and Ideas
- Spatial-Temporal Trade-off: By arranging frames in a grid, the model's self-attention mechanism can naturally capture temporal relationships by looking at different "cells" of the grid, effectively treating time as just another spatial dimension.
- Arrangement Matters: The authors tested different ways to stitch frames together. They found that a Matrix (Grid) layout significantly outperforms simple horizontal or vertical stacking. Stacking frames in a long strip (e.g., $1 \times 9$) distorts the visual information, whereas a $3 \times 3$ matrix preserves the aspect ratio and features of individual frames better.
- Constant Computational Cost: Perhaps the most striking finding is shown in their efficiency analysis. As the number of frames increases, the computational time for the "Single-Frame" approach remains almost perfectly flat, while the traditional "Multi-Frame" approach scales linearly.
Key Results
The "Single-Frame" approach proved that you don't need massive compute to reach the top of the leaderboard:
- Accuracy: On MSRVTT-MC, the model achieved 92.2% accuracy, slightly surpassing the previous state-of-the-art (92.1%) held by models like VideoCLIP and Singularity.
- Speed: The training and inference speed was approximately 3.8x to 3.9x faster than the multi-frame baseline.
- Efficiency: The model required only 30% of the memory compared to traditional methods, making it much more accessible for researchers with limited hardware (like a single NVIDIA GTX 3090).
Limitations and Future Work
While the results are impressive, the authors acknowledge a few hurdles:
- Video Length: The datasets used (MSRVTT and TrafficQA) consist of relatively short clips. It remains to be seen how well a grid-based approach scales to very long videos where $n^2$ might need to be very large, potentially leading to extremely high-resolution "mega-images" or loss of detail.
- Task Variety: The study focused strictly on VideoQA. Future work is needed to see if this "video-as-a-grid" approach holds up for other tasks like video retrieval or action recognition.
- Resolution Limits: There is a physical limit to how many frames you can cram into one image before the resolution of each individual frame becomes too low for the encoder to extract meaningful features.
Summary updated