Introduction
Visual Question Answering (VQA) was conceived to push computer vision beyond simple object recognition toward true "visual understanding"—the ability to reason about relationships, attributes, and causality within an image. To tackle this, the research community has developed increasingly complex architectures involving attention mechanisms, memory networks, and compositional logic. However, researchers from Facebook AI Research have raised a skeptical but necessary question: Are these complex models actually reasoning, or are they just exploiting dataset shortcuts?
This study investigates whether standard VQA datasets contain enough linguistic and statistical bias that a simple model, devoid of any complex reasoning machinery, can perform as well as or better than the state-of-the-art.
Methods: A Simple Binary Approach
The authors depart from the standard VQA approach of multi-class classification (predicting one answer from a fixed vocabulary). Instead, they propose a binary classification framework where the model evaluates a triplet: an image ($I$), a question ($Q$), and a candidate answer ($A$).
As shown in the architecture above, the model uses:
- Text Representation ($Q$ and $A$): A simple Bag-of-Words (BoW) approach, averaging pre-trained
word2vecembeddings. - Image Representation ($I$): Global features from a pre-trained ResNet-101, notably without any spatial attention.
- The Model: A Multi-Layer Perceptron (MLP) processes the concatenated features to output a score representing the probability that the $(I, Q, A)$ triplet is correct. During inference, the model scores all multiple-choice options and selects the highest.
Key Insights and Results
The study evaluated this approach on major benchmarks, including Visual7W Telling and VQA Real Multiple Choice. The results were surprising, showing that simple baselines often outperform complex "reasoning" models.
Major Findings:
- State-of-the-Art Performance: The MLP baseline achieved 67.1% accuracy on Visual7W, significantly outperforming previous models (62.2%) that utilized complex attention mechanisms. On the VQA dataset, it reached 65.2%, coming remarkably close to systems using sophisticated co-attention (66.1%).
- The Power of "Answer as Input": By including the answer in the input, the model exploits semantic similarities. For instance, it can learn that "morning" is a more plausible answer for a daytime scene than "blue," even with minimal visual processing.
- LSTMs vs. BoW: Surprisingly, simple BoW representations often performed as well as or better than LSTMs, suggesting that LSTMs may overfit on the relatively small scale of current VQA datasets.
- Dataset Bias: On "Why" questions, a text-only model (Q+A) achieved 64% accuracy, while adding the image only increased it to 68%. This highlights how much "common sense" and statistical bias are baked into the text of these benchmarks.
Limitations and Future Work
Despite its high performance, the authors acknowledge the limitations of their baseline:
- Lack of Spatial Reasoning: Using global image features makes the model struggle with questions requiring precise localization (e.g., "What is to the left of the red ball?").
- Counting: The model is poor at counting (56% accuracy), as global features often blur out small, individual objects.
The paper concludes that until datasets are designed to be less biased, it remains difficult to determine if a model is truly "understanding" an image or just performing sophisticated pattern matching. Future work should focus on visual grounding—explicitly linking words to specific image regions—rather than just building deeper "black box" classifiers.
Summary updated