Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, Ashish Sabharwal

Abstract

We propose a general framework called Text Modular Networks (TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models. To ensure solvability of simpler tasks, TMNs learn the textual input-output behavior (i.e., language) of existing models through their datasets. This differs from prior decomposition-based approaches which, besides being designed specifically for each complex task, produce decompositions independent of existing sub-models. Specifically, we focus on Question Answering (QA) and show how to train a next-question generator to sequentially produce sub-questions targeting appropriate sub-models, without additional human annotation. These sub-questions and answers provide a faithful natural language explanation of the model's reasoning. We use this framework to build ModularQA, a system that can answer multi-hop reasoning questions by decomposing them into sub-questions answerable by a neural factoid single-span QA model and a symbolic calculator. Our experiments show that ModularQA is more versatile than existing explainable systems for DROP and HotpotQA datasets, is more robust than state-of-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work.

Introduction

Modern NLP models excel at "single-hop" tasks—like finding a specific fact in a paragraph—but struggle with complex questions that require multi-step reasoning or discrete mathematics. Existing solutions generally fall into two camps: highly accurate but uninterpretable "blackbox" models, or modular models that require massive amounts of human-labeled decomposition data. Furthermore, existing modular approaches often produce decompositions that are mere fragments of the original text, failing when a sub-step requires a completely new phrasing, such as a mathematical operation.

Text Modular Networks (TMNs) bridge this gap by creating a system that "talks" to existing specialized models to solve complex problems while remaining fully transparent. By learning to communicate in the natural language "dialect" of these sub-models through distant supervision, the system provides faithful, human-readable explanations of its reasoning without requiring expensive, manually annotated decomposition data.

Methods

The core of the TMN framework is the MODULARQA system, which decomposes complex questions into a sequence of sub-tasks.

ModularQA decomposes complex questions into sub-questions answerable by specialized sub-models.

MODULARQA consists of three primary components:

  1. Sub-models ($A$): These are treated as blackboxes. The system utilizes a neural factoid QA model (RoBERTa trained on SQuAD) for information extraction and a symbolic calculator for mathematical operations.
  2. Next-Question Generator ($D$): A BART-based model that analyzes the original complex question and previous sub-answers to generate the next logical sub-question and select the appropriate sub-model.
  3. Distant Supervision via "Hints": Instead of human-labeled data, the authors "reverse-engineered" training data using "hints" (e.g., expected intermediate numbers). If a generated chain of sub-questions leads to the correct final answer, it is used to train the Next-Question Generator.

At inference time, the system employs a best-first search to explore potential reasoning chains, selecting the one that is most linguistically coherent and leads to a confident final answer.

Results

MODULARQA was evaluated on two major benchmarks: DROP (Discrete Reasoning Over Paragraphs) and HotpotQA (multi-hop reasoning).

Performance comparison of ModularQA against other interpretable and blackbox models on DROP and HotpotQA.

Performance and Robustness

On the DROP dataset, MODULARQA achieved an 87.9 F1 score, coming within 3.7% of the state-of-the-art blackbox model (NumNet+V2) while offering significantly higher interpretability. Notably, on the DROP Contrast Set—designed to test robustness against minimal phrasing changes—MODULARQA outperformed the state-of-the-art blackbox by 7-10%. This suggests that its reasoning-based approach is more stable than the pattern-matching typical of large transformers.

Interpretability and Human Trust

A key advantage of TMNs is that the explanation is the reasoning process. Because there are no hidden vector embeddings passed between modules—only natural language questions and answers—the "thought process" is inherently readable. In human evaluations, users found MODULARQA’s explanations more understandable than previous decomposition methods in 78% of cases, while also rating them higher for trustworthiness.

Discussion and Conclusion

TMNs demonstrate that language can serve as a universal interface between specialized modules. By forcing modules to communicate via text, the system's reasoning becomes faithful: if the explanation is incorrect, the answer is likely incorrect as well.

However, limitations remain. The current implementation is restricted to five specific reasoning types and cannot yet handle Boolean logic or list-based answers. Additionally, the best-first search used during inference is computationally more expensive than a single forward pass of a blackbox model. Future work may involve using Reinforcement Learning to enable faster, "greedy" decoding while maintaining the system's modular benefits.

Summary updated

Abstract

We propose a general framework called Text Modular Networks(TMNs) for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models. To ensure solvability of simpler tasks, TMNs learn the textual input-output behavior (i.e., language) of existing models through their datasets. This differs from prior decomposition-based approaches which, besides being designed specifically for each complex task, produce decompositions independent of existing sub-models. Specifically, we focus on Question Answering (QA) and show how to train a next-question generator to sequentially produce sub-questions targeting appropriate sub-models, without additional human annotation. These sub-questions and answers provide a faithful natural language explanation of the model's reasoning. We use this framework to build ModularQA, a system that can answer multi-hop reasoning questions by decomposing them into sub-questions answerable by a neural factoid single-span QA model and a symbolic calculator. Our experiments show that ModularQA is more versatile than existing explainable systems for DROP and HotpotQA datasets, is more robust than state-of-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work.
BibTeX
@inproceedings{khot2021text,
  author = {Tushar Khot and Daniel Khashabi and Kyle Richardson and Peter Clark and Ashish Sabharwal},
  title = {{Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=J9tOHcrho-7},
}

Loading related content...

Figure