Abstract
TVQA is a large scale video question answering (video-QA) dataset based on popular TV shows. The questions were specifically designed to require "both vision and language understanding to answer". In this work, we demonstrate an inherent bias in the dataset towards the textual subtitle modality. We infer said bias both directly and indirectly, notably finding that models trained with subtitles learn, on-average, to suppress video feature contribution. Our results demonstrate that models trained on only the visual information can answer ~45% of the questions, while using only the subtitles achieves ~68%. We find that a bilinear pooling based joint representation of modalities damages model performance by 9% implying a reliance on modality specific information. We also show that TVQA fails to benefit from the RUBi modality bias reduction technique popularised in VQA. By simply improving text processing using BERT embeddings with the simple model first proposed for TVQA, we achieve state-of-the-art results (72.13%) compared to the highly complex STAGE model (70.50%). We recommend a multimodal evaluation framework that can highlight biases in models and isolate visual and textual reliant subsets of data. Using this framework we propose subsets of TVQA that respond exclusively to either or both modalities in order to facilitate multimodal modelling as TVQA originally intended.
Introduction
Video Question Answering (Video-QA) tasks are designed to evaluate a model's ability to reason over both visual (video frames) and textual (subtitles/dialogue) information. The TVQA dataset was specifically crowdsourced to require "both vision and language understanding to answer." However, large-scale datasets often contain unintended biases that allow models to "cheat" by relying on unimodal shortcuts. This research investigates whether TVQA truly necessitates multimodal reasoning or if models are simply exploiting the dominant subtitle modality to achieve high performance without genuine video understanding.
Methodology
To dissect these biases, we developed a modular evaluation framework based on the original TVQA architecture. This framework allows for isolating and toggling different processing streams:
- Visual Features: ImageNet features, Regional features (Faster R-CNN), and Visual Concepts (object labels).
- Textual Features: Subtitles, Questions, and Answers.
We replaced standard GloVe word embeddings with contextual BERT embeddings to assess the impact of improved text processing. Crucially, we analyzed the pre-softmax answer votes to quantify the contribution of each modality to predictions.
Additionally, we tested techniques to encourage multimodal behavior, such as Bilinear Pooling for joint representations and RUBi (Reducing Unimodal Bias), a strategy effective in standard image VQA. We also introduced Modality Subsets for TVQA using an Inclusion-Exclusion Measure (IEM) to identify questions that strictly require multimodal information.
Key Insights
Our analysis revealed several critical findings regarding modality bias:
- Subtitles Suppress Vision: Subtitles are not merely more informative; the model actively learns to suppress video feature contributions when trained with subtitles. This suggests the model treats visual data as noise when strong textual cues are present.
- NLP Upgrades Outperform Complex Architectures: Swapping GloVe for BERT in a simple baseline model outperformed the complex, spatio-temporal STAGE model. This implies that previous state-of-the-art gains were likely due to superior text processing rather than better video-text fusion.
- Joint Representations and Bias Reduction Fail: Forcing joint representations via Bilinear Pooling dropped performance by 9%, and RUBi actually harmed accuracy. This indicates that TVQA's text biases are structural and resistant to standard regularization techniques used in other VQA tasks.
Experimental Results
The results confirm a severe unimodal dominance in the dataset. Subtitle-only models using BERT achieve 68.30% accuracy, while vision-only models max out at approximately 45%. By combining modalities (the SVI model) with BERT, we set a new state-of-the-art at 72.13%.
Our Inclusion-Exclusion Measure (IEM) analysis shows that only ~5.4% of questions can be uniquely answered by models without subtitles, and a mere ~3.8% of the validation set strictly requires multimodal information (i.e., answered correctly by the full model but missed by all unimodal models).
Conclusion
The TVQA dataset exhibits a significant bias toward textual subtitles, which often masks the lack of true multimodal reasoning in models. We recommend that future Video-QA research report performance on specific modality subsets to provide a clearer picture of model capabilities. Creating datasets that genuinely require multimodal integration without providing unimodal shortcuts remains a primary challenge for the field.
Summary updated