Revisiting Visual Question Answering Baselines

Allan Jabri, Armand Joulin, Laurens van der Maaten

Introduction

Visual Question Answering (VQA) was conceived to push computer vision beyond simple object recognition toward true "visual understanding"—the ability to reason about relationships, attributes, and causality within an image. To tackle this, the research community has developed increasingly complex architectures involving attention mechanisms, memory networks, and compositional logic. However, researchers from Facebook AI Research have raised a skeptical but necessary question: Are these complex models actually reasoning, or are they just exploiting dataset shortcuts?

This study investigates whether standard VQA datasets contain enough linguistic and statistical bias that a simple model, devoid of any complex reasoning machinery, can perform as well as or better than the state-of-the-art.

Methods: A Simple Binary Approach

The authors depart from the standard VQA approach of multi-class classification (predicting one answer from a fixed vocabulary). Instead, they propose a binary classification framework where the model evaluates a triplet: an image ($I$), a question ($Q$), and a candidate answer ($A$).

The proposed simple baseline architecture using Bag-of-Words (BoW) for text and CNN features for images.

As shown in the architecture above, the model uses:

  • Text Representation ($Q$ and $A$): A simple Bag-of-Words (BoW) approach, averaging pre-trained word2vec embeddings.
  • Image Representation ($I$): Global features from a pre-trained ResNet-101, notably without any spatial attention.
  • The Model: A Multi-Layer Perceptron (MLP) processes the concatenated features to output a score representing the probability that the $(I, Q, A)$ triplet is correct. During inference, the model scores all multiple-choice options and selects the highest.

Key Insights and Results

The study evaluated this approach on major benchmarks, including Visual7W Telling and VQA Real Multiple Choice. The results were surprising, showing that simple baselines often outperform complex "reasoning" models.

Accuracy comparison on the Visual7W Telling dataset across different question types.

Major Findings:

  • State-of-the-Art Performance: The MLP baseline achieved 67.1% accuracy on Visual7W, significantly outperforming previous models (62.2%) that utilized complex attention mechanisms. On the VQA dataset, it reached 65.2%, coming remarkably close to systems using sophisticated co-attention (66.1%).
  • The Power of "Answer as Input": By including the answer in the input, the model exploits semantic similarities. For instance, it can learn that "morning" is a more plausible answer for a daytime scene than "blue," even with minimal visual processing.
  • LSTMs vs. BoW: Surprisingly, simple BoW representations often performed as well as or better than LSTMs, suggesting that LSTMs may overfit on the relatively small scale of current VQA datasets.
  • Dataset Bias: On "Why" questions, a text-only model (Q+A) achieved 64% accuracy, while adding the image only increased it to 68%. This highlights how much "common sense" and statistical bias are baked into the text of these benchmarks.

Limitations and Future Work

Despite its high performance, the authors acknowledge the limitations of their baseline:

  1. Lack of Spatial Reasoning: Using global image features makes the model struggle with questions requiring precise localization (e.g., "What is to the left of the red ball?").
  2. Counting: The model is poor at counting (56% accuracy), as global features often blur out small, individual objects.

The paper concludes that until datasets are designed to be less biased, it remains difficult to determine if a model is truly "understanding" an image or just performing sophisticated pattern matching. Future work should focus on visual grounding—explicitly linking words to specific image regions—rather than just building deeper "black box" classifiers.

Summary updated

Abstract

Visual question answering (VQA) is an interesting learning setting for evaluating the abilities and shortcomings of current systems for image understanding. Many of the recently proposed VQA systems include attention or memory mechanisms designed to perform “reasoning”. Furthermore, for the task of multiple-choice VQA, nearly all of these systems train a multi-class classifier on image and question features to predict an answer. This paper questions the value of these common practices and develops a simple alternative model based on binary classification. Instead of treating answers as competing choices, our model receives the answer as input and predicts whether or not an image-question-answer triplet is correct. We evaluate our model on the Visual7W Telling and the VQA Real Multiple Choice tasks, and find that even simple versions of our model perform competitively. Our best model achieves state-of-the-art performance of $$65.8\,\%$$ accuracy on the Visual7W Telling task and compares surprisingly well with the most complex systems proposed for the VQA Real Multiple Choice task. Additionally, we explore variants of the model and study the transferability of the model between both datasets. We also present an error analysis of our best model, the results of which suggest that a key problem of current VQA systems lies in the lack of visual grounding and localization of concepts that occur in the questions and answers.
BibTeX
@inproceedings{jabri2016revisiting,
  author = {Allan Jabri and Armand Joulin and Laurens van der Maaten},
  title = {{Revisiting Visual Question Answering Baselines}},
  booktitle = {European Conference on Computer Vision},
  year = {2016},
  url = {https://openreview.net/forum?id=HJEq9qW_-H},
}
Figure