Abstract
Visual Question Answering (VQA) requires the effective integration of visual and linguistic modalities. Traditional models often fuse these features directly, which can lead to modality mismatch and the compounding of observation noise. This paper introduces Differential Networks (DN), a novel plug-and-play module that shifts the focus from raw feature values to the pairwise differences between feature elements. By modeling these differences, the proposed DN-based Fusion (DF) model effectively reduces noise and captures relational information, achieving state-of-the-art results across four major VQA benchmarks.
Introduction
The core challenge in VQA is the fusion step—combining visual features extracted from images with linguistic features from natural language questions. Existing models typically employ linear or bilinear operations for this task. However, these approaches face two significant hurdles: modality mismatch, where image and text features reside in disparate vector spaces, and observation noise, where raw features contain common noise components that are compounded during direct fusion.
Methodology: The Differential Network
To address these challenges, we propose the Differential Network (DN). Unlike a standard Fully-Connected Network (FCN) that focuses on individual feature elements (the diagonal of a weight matrix), the DN calculates the weighted sum of differences between all pairs of elements $(x_i - x_j)$.
This approach acts as a natural high-pass filter; if a feature contains a common noise component $\delta$, the subtraction $(v_i + \delta) - (v_j + \delta)$ effectively cancels it. Mathematically, while FCNs capture the "what" (the elements themselves), DNs capture the "how they differ" (the interactions). This allows the network to learn relational encoding within each modality before fusion occurs.
The DN-based Fusion (DF) Architecture
The complete architecture, termed DN-based Fusion (DF), follows a modular three-stage pipeline designed for efficiency and performance.
- Data Embedding: Visual objects are extracted using Faster-RCNN, while the question is encoded via a Gated Recurrent Unit (GRU).
- Differential Fusion: DN modules are applied to both visual and linguistic features. To manage the computational complexity of pairwise differences, we utilize low-rank matrix factorization to keep the model efficient.
- Decision Making: A final linear layer with a softmax activation predicts the answer from a predefined dictionary.
Experimental Results
The DF model was evaluated on four diverse benchmarks: VQA 1.0, VQA 2.0, COCO-QA, and TDIUC. It achieved state-of-the-art performance at the time of publication:
- VQA 1.0 & 2.0: Reached 68.62% and 68.59% accuracy respectively, significantly outperforming previous baselines.
- TDIUC: Demonstrated substantial gains in specific reasoning tasks, including a 5.12% increase in "Counting" and a 5.93% increase in "Color Attributes."
- Ablation Studies: Results confirmed that the differential operation is the primary driver of success, as replacing the DN module with standard fusion led to a significant drop in performance.
Conclusion
Differential Networks offer a robust solution for multi-modal feature fusion by prioritizing relative differences over raw values. While challenges remain in complex numerical reasoning (such as precise counting), the DN module's plug-and-play nature makes it a promising candidate for other tasks like image captioning and video-grounded dialogue. Future work will explore refining these differential operations for more sophisticated reasoning capabilities.
Summary updated