Abstract
Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical drawbacks: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability.
Introduction: The Scalability Wall in Visual Reasoning
Visual reasoning—answering complex questions about images like "What is the person in pink holding?"—requires a model to perform a sequence of logical steps. Neural Module Networks (NMNs) were designed for this using a "Lego-block" approach where different neural modules (e.g., a Filter[pink] module) are assembled into a program.
However, traditional NMNs face two significant hurdles. First is scalability: as tasks grow in complexity, requiring hundreds of unique modules, the model becomes bloated and difficult to train. Second is generalizability: if the model encounters a function it wasn't specifically trained for, it lacks the necessary "block" to handle it. The Meta Module Network (MMN) addresses these by shifting from a fixed inventory to a dynamic instantiation approach.
Methodology: Dynamic Instantiation via Meta Modules
The core of MMN is a single, "metamorphous" Meta Module. Instead of having separate weights for every possible function, this module receives a Function Recipe (a set of key-value pairs describing the operation). A Recipe Embedder turns this into a vector that parameterizes the Meta Module, effectively telling it how to behave for that specific reasoning step.
Architecture and Learning
The MMN pipeline involves several sophisticated components:
- Program Generation: Questions are parsed into logical steps (e.g.,
Select(ball)->Relate(beside, boy)). - Two-Layered Attention: The Meta Module uses dependency attention to understand context from previous modules and visual attention to extract information from the image.
- Teacher-Student Learning: To ensure the module learns the correct logic, a "Symbolic Teacher" uses ground-truth scene graphs to generate ideal attention maps. The "Student" Meta Module is trained to mimic these guidelines.
Experimental Results
The model was evaluated on the CLEVR and GQA datasets. GQA is particularly challenging, featuring real-world images and a diverse set of over 48 different functional semantics.
- State-of-the-Art Performance: MMN achieved 60.83% accuracy on GQA, significantly outperforming standard NMNs (55.70%) and even surpassing some large-scale pre-trained models like LXMERT.
- Zero-Shot Generalization: In experiments with held-out functions (like
filter_location), MMN achieved 77% accuracy without any training examples for those specific functions, while standard NMNs performed at a random-guess level (50%). This is possible because MMN leverages structural similarity in the embedding space to handle unseen operations.
Conclusion
By treating function specifications as "software" recipes rather than hard-coded architecture, the Meta Module Network provides a scalable and generalizable framework for visual reasoning. It maintains the interpretability of modular networks while overcoming the limitations of fixed inventories, representing a significant step forward in compositional AI.
Summary updated