Differential Networks for Visual Question Answering

Chenfei Wu, Jinlai Liu, Xiaojie Wang, Ruifan Li

Abstract

Visual Question Answering (VQA) requires the effective integration of visual and linguistic modalities. Traditional models often fuse these features directly, which can lead to modality mismatch and the compounding of observation noise. This paper introduces Differential Networks (DN), a novel plug-and-play module that shifts the focus from raw feature values to the pairwise differences between feature elements. By modeling these differences, the proposed DN-based Fusion (DF) model effectively reduces noise and captures relational information, achieving state-of-the-art results across four major VQA benchmarks.

Introduction

The core challenge in VQA is the fusion step—combining visual features extracted from images with linguistic features from natural language questions. Existing models typically employ linear or bilinear operations for this task. However, these approaches face two significant hurdles: modality mismatch, where image and text features reside in disparate vector spaces, and observation noise, where raw features contain common noise components that are compounded during direct fusion.

Methodology: The Differential Network

To address these challenges, we propose the Differential Network (DN). Unlike a standard Fully-Connected Network (FCN) that focuses on individual feature elements (the diagonal of a weight matrix), the DN calculates the weighted sum of differences between all pairs of elements $(x_i - x_j)$.

Comparison of DN and FCN architectures and their corresponding weight matrix structures.

This approach acts as a natural high-pass filter; if a feature contains a common noise component $\delta$, the subtraction $(v_i + \delta) - (v_j + \delta)$ effectively cancels it. Mathematically, while FCNs capture the "what" (the elements themselves), DNs capture the "how they differ" (the interactions). This allows the network to learn relational encoding within each modality before fusion occurs.

The DN-based Fusion (DF) Architecture

The complete architecture, termed DN-based Fusion (DF), follows a modular three-stage pipeline designed for efficiency and performance.

The complete Differential Fusion (DF) architecture, illustrating the data embedding, differential fusion, and decision-making stages.
  1. Data Embedding: Visual objects are extracted using Faster-RCNN, while the question is encoded via a Gated Recurrent Unit (GRU).
  2. Differential Fusion: DN modules are applied to both visual and linguistic features. To manage the computational complexity of pairwise differences, we utilize low-rank matrix factorization to keep the model efficient.
  3. Decision Making: A final linear layer with a softmax activation predicts the answer from a predefined dictionary.

Experimental Results

The DF model was evaluated on four diverse benchmarks: VQA 1.0, VQA 2.0, COCO-QA, and TDIUC. It achieved state-of-the-art performance at the time of publication:

  • VQA 1.0 & 2.0: Reached 68.62% and 68.59% accuracy respectively, significantly outperforming previous baselines.
  • TDIUC: Demonstrated substantial gains in specific reasoning tasks, including a 5.12% increase in "Counting" and a 5.93% increase in "Color Attributes."
  • Ablation Studies: Results confirmed that the differential operation is the primary driver of success, as replacing the DN module with standard fusion led to a significant drop in performance.

Conclusion

Differential Networks offer a robust solution for multi-modal feature fusion by prioritizing relative differences over raw values. While challenges remain in complex numerical reasoning (such as precise counting), the DN module's plug-and-play nature makes it a promising candidate for other tasks like image captioning and video-grounded dialogue. Future work will explore refining these differential operations for more sophisticated reasoning capabilities.

Summary updated

Abstract

The task of Visual Question Answering (VQA) has emerged in recent years for its potential applications. To address the VQA task, the model should fuse feature elements from both images and questions efficiently. Existing models fuse image feature element vi and question feature element qi directly, such as an element product viqi. Those solutions largely ignore the following two key points: 1) Whether vi and qi are in the same space. 2) How to reduce the observation noises in vi and qi. We argue that two differences between those two feature elements themselves, like (vi − vj) and (qi −qj), are more probably in the same space. And the difference operation would be beneficial to reduce observation noise. To achieve this, we first propose Differential Networks (DN), a novel plug-and-play module which enables differences between pair-wise feature elements. With the tool of DN, we then propose DN based Fusion (DF), a novel model for VQA task. We achieve state-of-the-art results on four publicly available datasets. Ablation studies also show the effectiveness of difference operations in DF model.
BibTeX
@inproceedings{wu2019differential,
  author = {Chenfei Wu and Jinlai Liu and Xiaojie Wang and Ruifan Li},
  title = {{Differential Networks for Visual Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33018997},
}
Figure