Standard deep learning models often operate as "black boxes," mapping inputs to outputs through monolithic architectures. While effective for pattern recognition, these models frequently struggle with systematicity—the ability to reason about novel combinations of known concepts. As A. Michael Carrell and Ian Berlot-Attwell demonstrate in Attribute Diversity Determines the Systematicity Gap in VQA, simply adding more data doesn't solve this; the gap is architectural.
To bridge this gap, a growing body of research advocates for modular decomposition. Instead of one giant network, these systems decompose complex tasks into a sequence of discrete, functional sub-tasks—essentially treating a natural language query as a program to be executed.
The Architecture of Decomposition
The core intuition behind this direction is that complex reasoning is often just the composition of simpler "primitives." A question like "How many red blocks are behind the cylinder?" can be broken down into a pipeline:
- Locate the cylinder.
- Shift the spatial focus to the area "behind" it.
- Filter for objects that are "red" and "blocks."
- Count the resulting set.
In this framework, researchers like Wei Ji and Junbin Xiao describe a shift from basic attention to neural-symbolic program decomposition. This process typically involves two stages: a planner (often a language model or parser) that translates the query into a structured program, and an executor that runs that program over the data.
From Neural Modules to Symbolic Programs
The "modules" in these systems vary in their implementation, ranging from fully learnable neural blocks to rigid symbolic functions.
Neural Module Networks
Early iterations, such as those discussed by Alane Suhr and Yoav Artzi, use specialized neural sub-networks that are dynamically assembled. Each module (e.g., a Find[red] module) is a small neural network trained to perform one specific operation. By chaining these, the model can handle diverse logical requirements while maintaining the differentiability needed for end-to-end training.
Neuro-Symbolic State Machines
Models like the Neural State Machine by Drew A. Hudson and Christopher D Manning take this further by simulating a differentiable automaton. They transform images into probabilistic scene graphs and questions into sequential instructions, allowing the model to "walk" through the logic of a scene step-by-step.
Program-Guided Execution
Recent work leverages the code-generation capabilities of Large Language Models (LLMs). Tanmay Gupta and Aniruddha Kembhavi introduced VisProg, which uses GPT-3 to generate Python-like programs. These programs orchestrate off-the-shelf models (like CLIP or object detectors) as modular subroutines. Similarly, GENOME by Zhenfang Chen and Chuang Gan allows an agent to autonomously create and reuse a library of these modules, mimicking how humans acquire and refine skills.
Why Modularity Works
The move toward modularity isn't just about performance; it addresses several fundamental weaknesses in deep learning:
- Data Efficiency: By reusing modules, models don't need to see every possible combination of objects and relations. Joy Hsu and Jiajun Wu show in NS3D that this approach achieves high accuracy with orders of magnitude less data than end-to-end baselines.
- Interpretability: Because the "reasoning" is an explicit program, users can see exactly which step failed. If a model fails to answer a question about a "red block," one can check if the
Find[red]module failed or if theCountmodule was the culprit. Hanwang Zhang and colleagues emphasize this in their work on eXplainable Neural Modules (XNMs), which disentangle perception from reasoning. - Algorithmic Robustness: Standard networks often fail at basic arithmetic or logic. By decomposing tasks into hierarchical procedures, as Roy Fox and Dawn Song demonstrate, models can achieve near-perfect accuracy on tasks like multi-digit multiplication that otherwise baffle LSTMs.
Expanding the Domain
While much of this research originated in Visual Question Answering, the modular program paradigm is expanding:
- Fact-Checking: Liangming Pan uses PROGRAMFC to decompose complex claims into verifiable sub-tasks.
- Robotics: Hamidreza Kasaei and Georgios Tziafas apply compositional reasoning to robot manipulation, allowing robots to follow unconstrained natural language instructions by executing "primitive" programs.
- Motion Analysis: Mark Endo and Joy Hsu extend this to temporal data with Modular Motion Programs, grounding complex human actions in 3D space.
By treating intelligence as the ability to orchestrate specialized skills rather than a single monolithic function, this research direction provides a path toward AI that is more transparent, efficient, and capable of the systematic reasoning that characterizes human thought.