Abstract
Joint vision and language tasks like visual question answering are fascinating because they explore high-level understanding, but at the same time, can be more prone to language biases. In this paper, we explore the biases in the MovieQA dataset and propose a strikingly simple model which can exploit them. We find that using the right word embedding is of utmost importance. By using an appropriately-trained word embedding, about half the Question-Answers (QAs) can be answered by looking at the questions and answers alone, completely ignoring narrative context from video clips, subtitles, and movie scripts. Compared to the best published papers on the leaderboard, our simple question+answer only model improves accuracy by 5% for video + subtitle category, 5% for subtitle, 15% for DVS and 6% higher for scripts.
Introduction
In the field of video question answering (VideoQA), benchmarks like MovieQA are designed to test a model's ability to integrate visual cues with narrative context. However, this research from Carnegie Mellon University and Argo AI reveals a significant "clever Hans" effect: many models may simply be exploiting language shortcuts rather than "understanding" the video.
The authors demonstrate that a strikingly simple text-only model—which ignores video and subtitles entirely—can outperform state-of-the-art multimodal models. This suggests that the way questions are authored in MovieQA introduces inherent language biases that allow for "visual" tasks to be solved through vocabulary matching.
Methods: A Strikingly Simple Model
The proposed model is intentionally "blind," ignoring every frame of video and every line of dialogue. Its architecture is straightforward:
- WikiWord Embeddings: A standard Word2Vec model is trained specifically on the Wikipedia movie plots associated with the MovieQA dataset.
- Sentence Representation: For a question and its five answer choices, the model computes sentence-level vectors via average-pooling.
- Similarity Scoring: These vectors pass through a single linear layer and are L2-normalized. The model calculates the dot product similarity between the question vector and each answer choice vector.
- Prediction: The answer choice with the highest similarity is selected.
The success of this model relies on the fact that MovieQA annotators wrote questions based on Wikipedia plot summaries without watching the movies, creating a direct semantic link between the question vocabulary and the correct answers.
Key Results and Insights
The "blind" model achieved remarkable results, ranking 1st in four out of five categories on the MovieQA leaderboard at the time of submission.
- Leaderboard Dominance: On the Video+Subtitle category, the QA-only model achieved 46.98% accuracy, significantly outperforming previous state-of-the-art multimodal attention networks.
- Specificity Trumps Scale: Generic embeddings (e.g., Google News) performed poorly. The model's power comes from being hyper-tuned to the specific movie plots in the dataset, capturing movie-specific semantics like linking "Quidditch" to "Harry Potter."
- Information Paradox: Interestingly, adding more movie plots to the training data beyond those in the dataset actually decreased performance, highlighting that the model is exploiting specific biases rather than general knowledge.
Discussion and Future Work
While the model performs exceptionally well on MovieQA, it fails on datasets like TVQA, where questions were authored by people watching the clips. This confirms that the model is not "better" at understanding movies but is instead a tool for exposing benchmark flaws.
The authors argue that future VideoQA datasets must be filtered using simple baseline models. If a Word2Vec lookup can solve a question, that question does not truly test multimodal understanding and should be removed to ensure the field continues to push the boundaries of genuine computer vision.
Summary updated