Learning to Compose Neural Networks for Question Answering

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein

Abstract

We describe a question answering model that applies to both images and structured knowledge bases. The model uses natural language strings to automatically assemble neural networks from a collection of composable modules. Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning, with only (world, question, answer) triples as supervision. Our approach, which we term a dynamic neural model network, achieves state-of-the-art results on benchmark datasets in both visual and structured domains.

Introduction

Traditional deep learning models for Question Answering (QA) typically employ a "one-size-fits-all" architecture, where data passes through a fixed sequence of layers regardless of the question's specific logic. However, human language is inherently compositional; a question like "What color is the cat?" requires different reasoning steps than "How many dogs are next to the fence?".

Researchers at UC Berkeley have developed Dynamic Neural Module Networks (D-NMNs) to address this. D-NMNs bridge the gap between classical semantic parsers, which excel at logic but struggle with "fuzzy" visual data, and standard neural networks, which handle vision well but often fail at complex multi-step reasoning. The system treats a neural network like a functional program, assembling specialized "modules" on the fly based on the linguistic structure of the input question.

Methodology: Dynamic Network Assembly

The D-NMN framework consists of two primary components trained jointly: a module inventory and a network layout predictor.

Overview of the D-NMN architecture, showing how a question is parsed into a layout and then assembled from a module inventory to query a knowledge source.

1. Module Inventory

The system maintains a collection of small, specialized neural network fragments. Each module is designed for a specific sub-task:

  • find[item]: Generates an attention map identifying the location of a specific item.
  • relate[in]: Shifts attention from one region to another (e.g., from a state to cities within it).
  • and: Computes the intersection of two attention maps.
  • describe[property]: Analyzes an attended region to produce a textual answer (e.g., "blue").
  • exists: A quantifier that determines if an attention map contains any active regions.

2. Network Layout Predictor

This component analyzes the input question using a dependency parser to understand its grammatical structure. An LSTM then proposes several "layouts"—blueprints for connecting the modules. Because selecting a layout is a discrete decision, the model is trained using Reinforcement Learning (REINFORCE) to identify layouts that yield correct answers. Simultaneously, standard backpropagation is used to train the internal weights of the individual modules.

Cross-Domain Application

A key strength of the D-NMN is its ability to handle both visual and structured data within a unified framework. By using differentiable logic, these modules return continuous distributions (attentions) rather than binary values, allowing the model to manage uncertainty in visual recognition.

Examples of D-NMN execution on visual (left) and structured (right) data, demonstrating how the same 'find' module logic applies across domains.

The model was evaluated on two distinct domains:

  • Visual Question Answering (VQA): Utilizing a dataset of over 200,000 natural images, the D-NMN achieved 59.4% accuracy, outperforming previous state-of-the-art attentional models.
  • GeoQA: In the geography domain, the model was tested on complex questions involving quantification (GeoQA+Q). It achieved 42.9% accuracy, a 20% relative improvement over fixed-structure baselines.

Discussion and Future Work

The D-NMN demonstrates that neural networks can learn to "look" at relevant parts of an image—such as attending to clothing when asked about color—without explicit spatial supervision. However, the model currently faces a "syntactic bottleneck," as it relies on off-the-shelf dependency parsers. If a question is ungrammatical or overly complex, the generated layouts may be suboptimal. Future research will focus on scaling to more deeply nested questions and improving the search strategy for network assembly as the module inventory grows.

Summary updated

Abstract

We describe a question answering model that applies to both images and structured knowledge bases. The model uses natural language strings to automatically assemble neural networks from a collection of composable modules. Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning, with only (world, question, answer) triples as supervision. Our approach, which we term a dynamic neural model network, achieves state-of-the-art results on benchmark datasets in both visual and structured domains.
BibTeX
@inproceedings{andreas2016learning,
  author = {Jacob Andreas and Marcus Rohrbach and Trevor Darrell and Dan Klein},
  title = {{Learning to Compose Neural Networks for Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=HyZqkQbu-H},
}
Figure