Abstract
Visual Question Answering (VQA) systems often struggle with language bias, frequently ignoring visual cues in favor of linguistic shortcuts. We present the Dual-Path Neural Module Network (DP-NMN), a framework designed to enforce genuine visual reasoning. By training on complementary image pairs and employing a pairwise reward mechanism, DP-NMN generates robust reasoning layouts. Integrated with a Region Proposal Network (RPN) for enhanced spatial awareness, the model demonstrates superior accuracy and interpretability on the VQA v2.0 and CLEVR benchmarks.
Introduction
Visual Question Answering (VQA) requires an AI to synthesize information from both text and images. However, many current models suffer from "language bias," where they learn to predict answers based on question patterns (e.g., answering "green" for any question about grass) without actually analyzing the image. While Neural Module Networks (NMNs) were proposed to address this by constructing modular reasoning programs, they often struggle with the complexity of real-world scenes and lack spatial reasoning capabilities.
The Dual-Path Neural Module Network (DP-NMN)
The DP-NMN addresses these challenges through three core architectural innovations designed to move beyond simple pattern matching toward structured reasoning.
1. Complementary Pair Training
Instead of processing single images, the DP-NMN is trained on "complementary pairs"—two distinct images associated with the same question. This setup forces the model to discover a reasoning layout that is valid across different visual contexts, preventing it from overfitting to specific image features.
2. Pairwise Reward System
The model utilizes a policy network to assemble a sequence of primitive modules (e.g., find, relocate, count). During training, a reinforcement learning approach is applied where a "pairwise reward" is granted only if the model correctly answers the question for both images in the pair. This stringent requirement encourages the generation of logical, generalizable reasoning paths.
3. Spatial Awareness via RPN
To handle complex spatial queries (e.g., "What is sitting beside the chair?"), the DP-NMN integrates a Region Proposal Network (RPN). The RPN identifies specific objects and their coordinates, allowing modules like relocate to perform precise spatial transformations rather than operating on a global image grid.
Experimental Results
The DP-NMN was evaluated on two major benchmarks:
- VQA v2.0: On this real-world dataset, DP-NMN achieved a single-model accuracy of 67.15% and an ensemble accuracy of 70.10%, outperforming standard NMN baselines and several attention-based models like Bottom-Up Top-Down (BUTD).
- CLEVR: On this synthetic reasoning task, the model reached 94.1% accuracy, demonstrating its ability to handle complex relational logic.
Ablation studies confirmed that the Pairwise Reward was the primary driver of performance; removing this component led to a significant drop in accuracy, highlighting the importance of the dual-path approach.
Interpretability and Reasoning
A key advantage of the DP-NMN is its inherent interpretability. Unlike "black box" models, it provides a transparent reasoning trace. For instance, when asked about the number of mirrors in a room, the model explicitly chains find and count modules.
The researchers observed that DP-NMN generated layouts that were approximately 12% larger than previous modular networks. This increased complexity allowed the model to navigate multi-step reasoning tasks that simpler architectures failed to resolve.
Conclusion
The Dual-Path Neural Module Network represents a significant step toward explainable and accurate visual reasoning. By leveraging complementary pairs and pairwise rewards, the framework effectively mitigates language bias. Future work will focus on scaling these modular approaches to even more complex, open-domain questions and reducing the dependency on structured complementary pairs.
Summary updated