Toward Accurate Visual Reasoning With Dual-Path Neural Module Networks

Ke Su, Hang Su, Jianguo Li, Jun Zhu

Abstract

Visual Question Answering (VQA) systems often struggle with language bias, frequently ignoring visual cues in favor of linguistic shortcuts. We present the Dual-Path Neural Module Network (DP-NMN), a framework designed to enforce genuine visual reasoning. By training on complementary image pairs and employing a pairwise reward mechanism, DP-NMN generates robust reasoning layouts. Integrated with a Region Proposal Network (RPN) for enhanced spatial awareness, the model demonstrates superior accuracy and interpretability on the VQA v2.0 and CLEVR benchmarks.

Introduction

Visual Question Answering (VQA) requires an AI to synthesize information from both text and images. However, many current models suffer from "language bias," where they learn to predict answers based on question patterns (e.g., answering "green" for any question about grass) without actually analyzing the image. While Neural Module Networks (NMNs) were proposed to address this by constructing modular reasoning programs, they often struggle with the complexity of real-world scenes and lack spatial reasoning capabilities.

The Dual-Path Neural Module Network (DP-NMN)

The DP-NMN addresses these challenges through three core architectural innovations designed to move beyond simple pattern matching toward structured reasoning.

1. Complementary Pair Training

Instead of processing single images, the DP-NMN is trained on "complementary pairs"—two distinct images associated with the same question. This setup forces the model to discover a reasoning layout that is valid across different visual contexts, preventing it from overfitting to specific image features.

2. Pairwise Reward System

The model utilizes a policy network to assemble a sequence of primitive modules (e.g., find, relocate, count). During training, a reinforcement learning approach is applied where a "pairwise reward" is granted only if the model correctly answers the question for both images in the pair. This stringent requirement encourages the generation of logical, generalizable reasoning paths.

The DP-NMN training pipeline, featuring the policy network, dual-path module execution, and the pairwise reward mechanism for gradient backpropagation.

3. Spatial Awareness via RPN

To handle complex spatial queries (e.g., "What is sitting beside the chair?"), the DP-NMN integrates a Region Proposal Network (RPN). The RPN identifies specific objects and their coordinates, allowing modules like relocate to perform precise spatial transformations rather than operating on a global image grid.

Experimental Results

The DP-NMN was evaluated on two major benchmarks:

  • VQA v2.0: On this real-world dataset, DP-NMN achieved a single-model accuracy of 67.15% and an ensemble accuracy of 70.10%, outperforming standard NMN baselines and several attention-based models like Bottom-Up Top-Down (BUTD).
  • CLEVR: On this synthetic reasoning task, the model reached 94.1% accuracy, demonstrating its ability to handle complex relational logic.

Ablation studies confirmed that the Pairwise Reward was the primary driver of performance; removing this component led to a significant drop in accuracy, highlighting the importance of the dual-path approach.

Interpretability and Reasoning

A key advantage of the DP-NMN is its inherent interpretability. Unlike "black box" models, it provides a transparent reasoning trace. For instance, when asked about the number of mirrors in a room, the model explicitly chains find and count modules.

Comparison of reasoning layouts between DP-NMN and standard End-to-End NMNs. DP-NMN generates more complex and accurate reasoning chains for multi-step questions.

The researchers observed that DP-NMN generated layouts that were approximately 12% larger than previous modular networks. This increased complexity allowed the model to navigate multi-step reasoning tasks that simpler architectures failed to resolve.

Conclusion

The Dual-Path Neural Module Network represents a significant step toward explainable and accurate visual reasoning. By leveraging complementary pairs and pairwise rewards, the framework effectively mitigates language bias. Future work will focus on scaling these modular approaches to even more complex, open-domain questions and reducing the dependency on structured complementary pairs.

Summary updated

Abstract

Visual reasoning is a critical stage in visual question answering (Antol et al., 2015), but most of the state-of-the-art methods categorized the VQA tasks as a classification problem without taking the reasoning process into account. Various approaches are proposed to solve this multi-modal task that requires both abilities of comprehension and reasoning. The recently proposed neural module network (Andreas et al., 2016b), which assembles the model with a few primitive modules, is capable of performing a spatial or arithmetical reasoning over the input image to answer the questions. Nevertheless, its performance is not satisfying especially in the real-world datasets (e.g., VQA 1.0& 2.0) due to its limited primitive modules and suboptimal layout. To address these issues, we propose a novel method of Dual-Path Neural Module Network which can implement complex visual reasoning by forming a more flexible layout regularized by the pairwise loss. Specifically, we first use the region proposal network to generate both visual and spatial information, which helps it perform spatial reasoning. Then, we advocate to process a pair of different images along with the same question simultaneously, named as a “complementary pair,” which encourages the model to learn a more reasonable layout by suppressing the overfitting to the language priors. The model can jointly learn the parameters in the primitive module and the layout generation policy, which is further boosted by introducing a no...
BibTeX
@article{su2020toward,
  author = {Ke Su and Hang Su and Jianguo Li and Jun Zhu},
  title = {{Toward Accurate Visual Reasoning With Dual-Path Neural Module Networks}},
  journal = {Frontiers Robotics AI},
  year = {2020},
  doi = {10.3389/frobt.2020.00109},
}
Figure