Abstract
While Visual Question Answering (VQA) models have achieved impressive accuracy on standard benchmarks, they often exhibit significant brittleness when faced with lexical variations. This paper explores whether pre-training—both within the language domain and across modalities—can mitigate this sensitivity. By evaluating models on the VQA-Rephrasings dataset, the researchers find that pre-trained components like SBERT and LXMERT provide substantial gains in robustness, although a performance gap between original and rephrased questions persists.
The Challenge of Lexical Robustness
In the current landscape of VQA, models are approaching human-level performance on datasets like VQA 2.0. However, these models frequently suffer from a lack of lexical robustness. A simple change in phrasing—for instance, from "Is that a fire truck?" to "Do you know whether or not that is a fire truck?"—can cause a model's confidence to plummet or its answer to change entirely.
This sensitivity poses a major hurdle for real-world deployment, where users naturally employ diverse linguistic styles. The authors investigate whether this issue stems from model architecture or a lack of linguistic variety in training data, specifically testing if pre-training can bridge this "rephrasing gap."
Methods: Leveraging Pre-trained Representations
The study compares standard VQA architectures against versions enhanced with pre-trained components:
- The Baseline (BUTD): The "Bottom-Up and Top-Down" attention model using a standard Gated Recurrent Unit (GRU) to encode questions.
- Language Pre-training (BUTD + SBERT): Replacing the GRU with Sentence-BERT (SBERT), a transformer-based encoder pre-trained on massive text corpora to produce semantically rich embeddings.
- Cross-modal Pre-training (LXMERT): A transformer-based framework designed for vision-and-language tasks, tested in various states of pre-training (random initialization, BERT weights only, and fully pre-trained).
Evaluation was conducted using the VQA-Rephrasings (VQA-R) dataset, which pairs original VQA 2.0 questions (ORI) with three human-written rephrasings (REP) for each image.
Key Insights: Semantic Stability and Keyword Latching
A core finding of the research is the superior semantic stability of transformer-based pre-trained models compared to traditional recurrent units.
As shown in the embedding analysis, SBERT maps a question and its rephrasings to nearly the same vector space (91% cosine similarity), whereas the standard GRU's embeddings for the same pairs only achieve 60% similarity. This suggests that pre-training helps models ignore the "noise" of syntactic variation and focus on core semantic content—a phenomenon the authors call "keyword latching."
Furthermore, the researchers found that complex architectures like LXMERT perform no better than simpler models without pre-training. The robustness to lexical variation is primarily a product of the pre-training phase rather than the transformer layers themselves. While language-only pre-training (BERT) is beneficial, cross-modal pre-training (LXMERT) proves superior by aligning visual entities with linguistic concepts.
Results and the Persistent Performance Gap
The experimental results highlight the power of pre-training:
- SBERT Boost: Swapping the GRU for SBERT in the BUTD model improved accuracy on rephrased questions by 5.42% relative to the baseline.
- LXMERT Dominance: Fully pre-trained LXMERT achieved state-of-the-art results, showing a 20.95% relative gain over its non-pre-trained counterpart on rephrased questions.
Despite these significant improvements, a "residual gap" remains. Even the best-performing model (LXMERT) experienced a ~10% drop in accuracy when moving from original questions (73.61%) to rephrased ones (66.27%).
Conclusion and Future Directions
While pre-training significantly enhances the lexical robustness of VQA models, it does not entirely solve the problem. The performance penalty for rephrased questions suggests that current pre-training methods still struggle with the distribution shift between standard training sets and the variety of natural human phrasing. Future work may need to investigate why certain answer types, such as numerical questions, remain particularly difficult to stabilize and how to further bridge the gap between original and rephrased queries.
Summary updated