Meta Module Network for Compositional Visual Reasoning

Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Yang Wang, Jingjing Liu

Abstract

Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical drawbacks: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability. Experiments on GQA and CLEVR datasets validate the superiority of MMN over state-of-the-art NMN designs.

Introduction

Visual reasoning—answering complex questions about images—requires both visual perception and logical composition. Traditional approaches often fall into two categories: monolithic models that process the entire image and question through a single deep network, and Neural Module Networks (NMNs) that parse questions into functional programs executed by specialized neural modules.

While NMNs offer superior interpretability, they face significant hurdles in scalability and generalizability. A standard NMN requires a unique neural network for every possible function (e.g., "filter color," "compare size"). As the complexity of datasets like GQA grows, maintaining a massive inventory of specialized modules becomes computationally prohibitive. Furthermore, these models struggle with "zero-shot" scenarios; if a model has never seen a "filter square" module during training, it cannot reason about squares at test time.

Comparison of Monolithic, NMN, and the proposed MMN architectures. MMN uses a central Meta Module to dynamically instantiate modules based on function recipes.

The Meta Module Network (MMN) Framework

To overcome these limitations, we introduce the Meta Module Network (MMN). The core innovation is the replacement of a rigid library of task-specific modules with a single, flexible "Meta Module." This Meta Module accepts a function recipe—a vector representation of the function's semantics and arguments—and dynamically morphs its internal parameters to perform the required operation.

Architectural Overview

The MMN framework consists of three primary stages:

  1. Program Generation: A semantic parser converts natural language questions into functional programs. This process is hierarchical, first generating a "sketch" of operations and then filling in specific arguments.
  2. Visual Encoding: The system uses a pre-trained object detector to extract regional features. These features are refined through self-attention and question-guided cross-attention to ensure the model focuses on relevant visual cues.
  3. Meta Module Instantiation: The Meta Module uses a two-layered attention mechanism to process visual features and outputs from previous modules. By conditioning on the function recipe, it effectively reconfigures its logic on the fly, allowing it to act as a "filter," "relate," or "compare" module within the same parameter space.
The MMN framework: (1) Program Generator parses the question; (2) Meta Module instantiates specific modules; (3) Execution graph processes visual features; (4) Teacher-Student framework provides supervision.

Teacher-Student Training

To ensure the Meta Module learns the correct reasoning logic, we employ a Teacher-Student framework. A "Symbolic Teacher" utilizes the ground-truth scene graph of an image to determine the ideal attention distribution at each reasoning step. This provides intermediate "guidelines" to the "Student" (the Meta Module), helping it align its visual attention with the logical requirements of the program.

Key Insights and Results

The MMN approach offers several distinct advantages:

  • Functions as Embeddings: By representing functions in a continuous embedding space, the model exploits structural similarities. This enables zero-shot generalization; for example, if the model understands "filter color," it can generalize to "filter shape" because their recipes share a similar structure.
  • Parameter Efficiency: Model complexity remains constant regardless of the number of functions, as all instance modules share the parameters of the central Meta Module.
  • State-of-the-Art Performance: On the GQA benchmark, MMN achieved 60.83% accuracy, outperforming standard NMNs and even large-scale pre-trained models like LXMERT. In zero-shot experiments on held-out functions, MMN maintained high accuracy (61%), whereas traditional NMNs would be limited to random guessing.

Conclusion

Meta Module Networks represent a significant step toward scalable and generalizable compositional reasoning. By moving from discrete modules to a continuous "metamorphous" design, MMN retains the interpretability of modular approaches while gaining the flexibility of end-to-end systems. Future research will focus on improving visual feature quality for complex spatial relations and reducing the dependency on ground-truth scene graphs for training.

Summary updated

Abstract

Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical drawbacks: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability. Experiments on GQA and CLEVR datasets validate the superiority of MMN over state-of-the-art NMN designs. Synthetic experiments on held-out unseen functions from GQA dataset also demonstrate the strong generalizability of MMN. Our code and model are released in Github https://github.com/wenhuchen/Meta-Module-Network.
BibTeX
@inproceedings{chen2021meta,
  author = {Wenhu Chen and Zhe Gan and Linjie Li and Yu Cheng and William Yang Wang and Jingjing Liu},
  title = {{Meta Module Network for Compositional Visual Reasoning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=-UQoARQufnl},
}
Figure