Feature Enhancement in Attention for Visual Question Answering

Yuetan Lin, Zhangyang Pang, Donghui Wang, Yueting ZhuangIJCAI 2018 · DOI

TL;DR

This paper tackles the "semantic gap" between visual and textual features in Visual Question Answering (VQA) models. They propose a novel "feature enhancement" approach to improve the attention mechanism by enriching region representations with semantic information, multi-level visual features, and refined semantic details, leading to significant accuracy gains.

What Problem Does This Paper Address?

Visual Question Answering (VQA) models aim to answer natural language questions about images. A crucial component of these models is the attention mechanism, which helps the model focus on relevant parts of the image and question. However, existing VQA models often struggle because the visual features (from image processing) and textual features (from language processing) come from different sources and have an inherent "semantic gap." This inconsistency can hinder the attention mechanism's ability to accurately correlate image regions with question words, leading to suboptimal performance. The paper aims to bridge this gap by enhancing the representations used within the attention mechanism.

Methods

The core of this paper is the proposed "feature enhancement" mechanism for attention in VQA models. This mechanism operates on three key aspects:

  • Region Semantic Representation (T/V): Instead of solely relying on visual features for image regions, the authors propose using semantic representations (like object labels predicted from the region) that are in the same word embedding space as the question. This makes the region representation more directly comparable to the question's semantic content.
  • Representation Enrichment using Multi-Level Features (TV/V, TV/TV): To capture richer information, they combine semantic representations with visual features from different hierarchical levels of the CNN. This allows the model to leverage both abstract semantic understanding and fine-grained visual details (e.g., color, shape).
  • Semantic Representation Refinement (rTV/V, rTV/rTV): Recognizing that a single semantic label might be insufficient, they refine the semantic representation by considering multiple predicted object labels and their probabilities. This makes the semantic representation more robust and informative.

The paper builds upon the "bottom-up attention" model, which uses region proposals from Faster R-CNN. Their proposed enhancements are integrated into this framework, affecting how region features are used to generate attention weights and how features are fused.

Datasets

The primary dataset used for training and evaluation is VQA v2.0. The authors note that VQA v2.0 is balanced to avoid answer bias and is larger than its predecessor. They specifically evaluate on the "real images" subset of VQA v2.0, which is more representative of real-world applications. They explicitly state that their model is trained without additional data from sources like Visual Genome.

Key Insights and Ideas

  • Bridging the Semantic Gap is Crucial: The paper strongly demonstrates that the inconsistency between visual and textual features is a significant bottleneck for attention mechanisms in VQA.
  • Semantics Enhance Attention: Using semantic representations of image regions (even just object labels) directly improves the attention mechanism's ability to correlate with the question, leading to better VQA performance.
  • Multi-Level Features Offer Complementarity: Combining semantic and visual features from different levels of a CNN provides a more comprehensive representation, capturing both abstract concepts and concrete visual details.
  • Refinement Improves Robustness: Refining semantic representations by considering multiple labels and their probabilities makes them more robust and less prone to errors from a single, potentially incorrect, label.
  • Cumulative Benefits: Integrating all three feature enhancement mechanisms (semantic representation, multi-level enrichment, and refinement) leads to the best overall performance, suggesting a synergistic effect.

Key Results

The paper reports significant accuracy improvements on the VQA v2.0 validation set using their proposed feature enhancement mechanisms.

  • Base-att model:
    • V/V (conventional attention): 61.07% overall accuracy.
  • rTV/rTV (fully enhanced model): 62.26% overall accuracy.
  • This represents a gain of 1.19% accuracy by integrating all proposed enhancements.
  • Double-att model:
    • Achieved state-of-the-art single model accuracy on the VQA v2.0 test-dev and test-std splits without using additional Visual Genome data.
  • Their ensemble model ranked second on the leaderboard with 70.40% overall accuracy.
  • Qualitative Results: Visualizations show that their enhanced attention mechanisms focus more accurately on relevant image regions (e.g., kite and kayak) and lead to correct answers, whereas conventional attention sometimes diverges or misses critical areas.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that certain question types, such as counting, direction-based questions, and optical character recognition (OCR), remain challenging.
  • The model's performance on "frog" detection in one qualitative example was poor, speculated to be due to the detector not being trained on that specific object.
  • Future Work:
    • The authors suggest integrating more diverse region representations (e.g., activities, attributes, relationships).
  • They propose incorporating specialized modules for tackling specific challenging question types.
  • The general idea of combining multi-level features in attention-based methods is presented as applicable to other multi-modal tasks.

Summary updated

BibTeX
@inproceedings{lin2018feature,
  author = {Yuetan Lin and Zhangyang Pang and Donghui Wang and Yueting Zhuang},
  title = {{Feature Enhancement in Attention for Visual Question Answering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/586},
}
Figure