Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, Ashish Sabharwal

Abstract

Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, Ashish Sabharwal. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2021.

Introduction

Modern AI models often struggle with "multi-hop" reasoning—tasks that require combining multiple pieces of information or performing discrete operations (like math) to reach an answer. While "black-box" neural networks can be trained to solve these, they are notoriously uninterpretable; we don't know how they arrived at an answer.

Previous attempts to build modular or decomposition-based systems faced two major hurdles: the Annotation Bottleneck, requiring expensive human-annotated "blueprints" for every complex question, and the Language Gap, where a decomposition model might generate a sub-question that makes sense to a human but is phrased in a way the specific sub-model (e.g., a SQuAD-trained QA model) doesn't understand.

The authors propose Text Modular Networks (TMNs), a framework that learns to decompose tasks automatically and ensures those decompositions are "in-scope" for the models being used.

Examples of MODULARQA decomposing complex questions into sub-tasks for different datasets.

Methods

The authors introduce a specific implementation called MODULARQA. The architecture consists of two main parts:

  • Sub-models ($A$): Existing "black-box" tools, such as a neural QA model (RoBERTa-Large trained on SQuAD) and a symbolic calculator.
  • Next-Question Generator ($D$): A BART-based model that looks at the complex question and any previous sub-questions/answers to decide the next sub-question and the appropriate sub-model.

The "Secret Sauce": Distant Supervision

To train the generator without human-labeled decompositions, the authors use distant supervision. They identify "hints" (intermediate numbers or entities) that could lead to the final answer. A separate "sub-task question model" generates questions that would result in those hints, ensuring the generator learns to produce questions that the sub-models are actually capable of answering—effectively learning to speak the sub-model's "language."

Inference

During prediction, the system uses a best-first search to explore multiple possible decomposition chains. It uses a scoring function to pick the most plausible path, ensuring the sub-questions use words from the original query and lead to a confident final answer.

Results

The system was evaluated on two major benchmarks: DROP (numerical reasoning) and HotpotQA (multi-hop reasoning).

Performance comparison of MODULARQA against other models on DROP and HotpotQA benchmarks.
  • Competitive Performance: On DROP, MODULARQA achieved an 87.9 F1 score, only 3.7% behind the state-of-the-art black-box model (NumNet+V2), while providing full interpretability.
  • Superior Robustness: On the DROP "contrast set" (adversarial perturbations), MODULARQA outperformed the state-of-the-art black-box model by 7% F1.
  • Sample Efficiency: When trained on only 20% of the available data, MODULARQA outperformed black-box models, suggesting its modular structure provides a powerful inductive bias.
  • Human Preference: Human evaluators found MODULARQA’s explanations more trustworthy and understandable than prior decomposition models in approximately 70% of cases.

Conclusion

Text Modular Networks demonstrate that complex reasoning can be achieved by orchestrating existing models through natural language. By learning to decompose tasks into the specific input distribution of sub-models, MODULARQA provides faithful, human-readable explanations and superior robustness compared to traditional black-box systems. Future work includes expanding the reasoning scope to tasks like counting and sorting, and optimizing the search process for faster inference.

Summary updated

BibTeX
@inproceedings{khot2021text,
  author = {Tushar Khot and Daniel Khashabi and Kyle Richardson and Peter Clark and Ashish Sabharwal},
  title = {{Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=J9tOHcrho-7},
}
Figure