Compositional Reasoning with Modular Neural Programs

This direction is split into other directions: dynamic neural module assembly. Its last version stays readable and citable here.

Version 1 of 1201 papers225 conceptsfrozen 2026-10-04

Standard deep learning models often operate as "black boxes," mapping inputs to outputs through monolithic architectures. While effective for pattern recognition, these models frequently struggle with systematicity—the ability to reason about novel combinations of known concepts. As A. Michael Carrell and Ian Berlot-Attwell demonstrate in Attribute Diversity Determines the Systematicity Gap in VQA, simply adding more data doesn't solve this; the gap is architectural.

To bridge this gap, a growing body of research advocates for modular decomposition. Instead of one giant network, these systems decompose complex tasks into a sequence of discrete, functional sub-tasks—essentially treating a natural language query as a program to be executed.

The Architecture of Decomposition

The core intuition behind this direction is that complex reasoning is often just the composition of simpler "primitives." A question like "How many red blocks are behind the cylinder?" can be broken down into a pipeline:

  1. Locate the cylinder.
  2. Shift the spatial focus to the area "behind" it.
  3. Filter for objects that are "red" and "blocks."
  4. Count the resulting set.

In this framework, researchers like Wei Ji and Junbin Xiao describe a shift from basic attention to neural-symbolic program decomposition. This process typically involves two stages: a planner (often a language model or parser) that translates the query into a structured program, and an executor that runs that program over the data.

From Neural Modules to Symbolic Programs

The "modules" in these systems vary in their implementation, ranging from fully learnable neural blocks to rigid symbolic functions.

Neural Module Networks

Early iterations, such as those discussed by Alane Suhr and Yoav Artzi, use specialized neural sub-networks that are dynamically assembled. Each module (e.g., a Find[red] module) is a small neural network trained to perform one specific operation. By chaining these, the model can handle diverse logical requirements while maintaining the differentiability needed for end-to-end training.

Neuro-Symbolic State Machines

Models like the Neural State Machine by Drew A. Hudson and Christopher D Manning take this further by simulating a differentiable automaton. They transform images into probabilistic scene graphs and questions into sequential instructions, allowing the model to "walk" through the logic of a scene step-by-step.

Program-Guided Execution

Recent work leverages the code-generation capabilities of Large Language Models (LLMs). Tanmay Gupta and Aniruddha Kembhavi introduced VisProg, which uses GPT-3 to generate Python-like programs. These programs orchestrate off-the-shelf models (like CLIP or object detectors) as modular subroutines. Similarly, GENOME by Zhenfang Chen and Chuang Gan allows an agent to autonomously create and reuse a library of these modules, mimicking how humans acquire and refine skills.

Why Modularity Works

The move toward modularity isn't just about performance; it addresses several fundamental weaknesses in deep learning:

  • Data Efficiency: By reusing modules, models don't need to see every possible combination of objects and relations. Joy Hsu and Jiajun Wu show in NS3D that this approach achieves high accuracy with orders of magnitude less data than end-to-end baselines.
  • Interpretability: Because the "reasoning" is an explicit program, users can see exactly which step failed. If a model fails to answer a question about a "red block," one can check if the Find[red] module failed or if the Count module was the culprit. Hanwang Zhang and colleagues emphasize this in their work on eXplainable Neural Modules (XNMs), which disentangle perception from reasoning.
  • Algorithmic Robustness: Standard networks often fail at basic arithmetic or logic. By decomposing tasks into hierarchical procedures, as Roy Fox and Dawn Song demonstrate, models can achieve near-perfect accuracy on tasks like multi-digit multiplication that otherwise baffle LSTMs.

Expanding the Domain

While much of this research originated in Visual Question Answering, the modular program paradigm is expanding:

By treating intelligence as the ability to orchestrate specialized skills rather than a single monolithic function, this research direction provides a path toward AI that is more transparent, efficient, and capable of the systematic reasoning that characterizes human thought.

Papers in this version

Showing 60 of 201 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_reasoning_with_modular_neu_dd025be7,
  title = {Compositional Reasoning with Modular Neural Programs},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-reasoning-with-modular-neural-programs/rdr_dd025be734ca49b7a04ce696d6be2c22},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_reasoning_with_modular_neu_dd025be7_v1,
  title = {Compositional Reasoning with Modular Neural Programs},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-reasoning-with-modular-neural-programs/rdr_dd025be734ca49b7a04ce696d6be2c22/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}