TL;DR
This paper exposes a severe modality bias in the TVQA (Video Question Answering) dataset, revealing that models rely almost entirely on textual subtitles rather than multimodal video cues. By simply upgrading the text embeddings to BERT in a basic model, the authors achieve state-of-the-art results, proving that complex multimodal architectures are often just masking strong unimodal text biases.
What Problem Does This Paper Address?
Video Question Answering (Video-QA) tasks are designed to test a model's ability to reason over both visual (video frames) and textual (subtitles/dialogue) information. The TVQA dataset was specifically crowdsourced to require "both vision and language understanding to answer." However, large datasets often contain unintended biases that allow models to "cheat" by relying on shortcuts. The authors investigate whether TVQA actually requires multimodal reasoning, or if models are simply exploiting one dominant modality (like the subtitles) to guess the correct answers without truly understanding the video.
Methods
To dissect the dataset's biases, the authors built a modular evaluation framework based on the original TVQA architecture. This allowed them to isolate and toggle different processing streams:
- Visual Features: ImageNet features, Regional features (Faster R-CNN), and Visual Concepts (object labels).
- Textual Features: Subtitles, Questions, and Answers.
They replaced standard GloVe word embeddings with contextual BERT embeddings to see how improved text processing affected performance. Crucially, they analyzed the pre-softmax answer votes to quantify exactly how much each modality contributed to true and false positive predictions.
Finally, they tested two techniques to see if they could force better multimodal behavior: Bilinear Pooling (to create a joint representation of text and video) and RUBi (Reducing Unimodal Bias, a learning strategy popular in standard image VQA).
Datasets
- TVQA: The primary focus of the paper. A large-scale Video-QA dataset based on popular TV shows, containing over 150k QA pairs with timestamps for video and subtitles.
- VQA-CP v2 & VQA v2: Used briefly as benchmarks to test the RUBi bias-reduction strategy.
- Novel Contribution: The authors introduce Modality Subsets for TVQA using an Inclusion-Exclusion Measure (IEM). This splits the dataset into subsets of questions that can be answered by text-only models, video-only models, or strictly require multimodal models.
Key Insights and Ideas
- Subtitles actively suppress vision: It is not just that subtitles are more informative. The pre-softmax vote analysis revealed that when trained with subtitles, the model learns to actively suppress the contribution of video features when predicting the correct answer.
- NLP upgrades > Multimodal architecture: Simply swapping GloVe for BERT in a structurally simple baseline model beat the highly complex, spatio-temporal STAGE model. This implies that previous state-of-the-art gains on TVQA were likely due to better text processing, not superior video-text fusion.
- Joint representations hurt: Forcing a joint representation of text and video via Bilinear Pooling dropped performance by 9%. This suggests the dataset doesn't reward true multimodal semantics; modality-specific shortcuts are too strong.
- Standard bias reduction fails: RUBi, which successfully mitigates language priors in standard image-VQA, actually harms TVQA performance. This indicates that TVQA's text biases are deeply structural to the task, rather than just simple language priors that can be regularized away.
Key Results
- Unimodal Dominance: Subtitle-only models using BERT achieve 68.30% accuracy. In contrast, vision-only models max out at around ~45% accuracy.
- New SOTA: Combining subtitles and vision (the SVI model) with BERT achieves 72.13%, setting a new state-of-the-art and beating the complex STAGE model (70.50%).
- Lack of Multimodality: Only ~5.4% of questions in the validation set can be uniquely answered by models that don't use subtitles.
- The Multimodal Subset: Using their Inclusion-Exclusion Measure, the authors found that only ~3.8% of the validation set strictly requires multimodal information (i.e., answered correctly by the full model but missed by all unimodal models).
Limitations and Future Work
The authors acknowledge that their proposed Inclusion-Exclusion Measure (IEM) subsets are inherently tied to the specific models evaluated. As better unimodal models are developed in the future, the "multimodal-only" subset will likely shrink even further.
For future work, the authors strongly recommend that Video-QA researchers evaluate and report performance on specific modality subsets rather than just overall accuracy. They also caution that creating genuinely multimodal datasets without introducing unimodal shortcuts remains a massive, unresolved challenge for the field.
Summary updated