Can Pre-training help VQA with Lexical Variations?

Shailza Jolly, Shubham KapoorEMNLP 2020 · DOI

Abstract

While Visual Question Answering (VQA) models have achieved impressive accuracy on standard benchmarks, they often exhibit significant brittleness when faced with lexical variations. This paper explores whether pre-training—both within the language domain and across modalities—can mitigate this sensitivity. By evaluating models on the VQA-Rephrasings dataset, the researchers find that pre-trained components like SBERT and LXMERT provide substantial gains in robustness, although a performance gap between original and rephrased questions persists.

The Challenge of Lexical Robustness

In the current landscape of VQA, models are approaching human-level performance on datasets like VQA 2.0. However, these models frequently suffer from a lack of lexical robustness. A simple change in phrasing—for instance, from "Is that a fire truck?" to "Do you know whether or not that is a fire truck?"—can cause a model's confidence to plummet or its answer to change entirely.

Examples of how minor lexical variations in questions can lead to drastic changes in VQA model predictions and confidence.

This sensitivity poses a major hurdle for real-world deployment, where users naturally employ diverse linguistic styles. The authors investigate whether this issue stems from model architecture or a lack of linguistic variety in training data, specifically testing if pre-training can bridge this "rephrasing gap."

Methods: Leveraging Pre-trained Representations

The study compares standard VQA architectures against versions enhanced with pre-trained components:

  1. The Baseline (BUTD): The "Bottom-Up and Top-Down" attention model using a standard Gated Recurrent Unit (GRU) to encode questions.
  2. Language Pre-training (BUTD + SBERT): Replacing the GRU with Sentence-BERT (SBERT), a transformer-based encoder pre-trained on massive text corpora to produce semantically rich embeddings.
  3. Cross-modal Pre-training (LXMERT): A transformer-based framework designed for vision-and-language tasks, tested in various states of pre-training (random initialization, BERT weights only, and fully pre-trained).

Evaluation was conducted using the VQA-Rephrasings (VQA-R) dataset, which pairs original VQA 2.0 questions (ORI) with three human-written rephrasings (REP) for each image.

Key Insights: Semantic Stability and Keyword Latching

A core finding of the research is the superior semantic stability of transformer-based pre-trained models compared to traditional recurrent units.

Distribution of cosine similarities between original and rephrased question embeddings, comparing SBERT and GRU encoders.

As shown in the embedding analysis, SBERT maps a question and its rephrasings to nearly the same vector space (91% cosine similarity), whereas the standard GRU's embeddings for the same pairs only achieve 60% similarity. This suggests that pre-training helps models ignore the "noise" of syntactic variation and focus on core semantic content—a phenomenon the authors call "keyword latching."

Furthermore, the researchers found that complex architectures like LXMERT perform no better than simpler models without pre-training. The robustness to lexical variation is primarily a product of the pre-training phase rather than the transformer layers themselves. While language-only pre-training (BERT) is beneficial, cross-modal pre-training (LXMERT) proves superior by aligning visual entities with linguistic concepts.

Results and the Persistent Performance Gap

The experimental results highlight the power of pre-training:

  • SBERT Boost: Swapping the GRU for SBERT in the BUTD model improved accuracy on rephrased questions by 5.42% relative to the baseline.
  • LXMERT Dominance: Fully pre-trained LXMERT achieved state-of-the-art results, showing a 20.95% relative gain over its non-pre-trained counterpart on rephrased questions.

Despite these significant improvements, a "residual gap" remains. Even the best-performing model (LXMERT) experienced a ~10% drop in accuracy when moving from original questions (73.61%) to rephrased ones (66.27%).

Conclusion and Future Directions

While pre-training significantly enhances the lexical robustness of VQA models, it does not entirely solve the problem. The performance penalty for rephrased questions suggests that current pre-training methods still struggle with the distribution shift between standard training sets and the variety of natural human phrasing. Future work may need to investigate why certain answer types, such as numerical questions, remain particularly difficult to stabilize and how to further bridge the gap between original and rephrased queries.

Summary updated

BibTeX
@inproceedings{jolly2020can,
  author = {Shailza Jolly and Shubham Kapoor},
  title = {{Can Pre-training help VQA with Lexical Variations?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.257},
}
Figure