Abstract
Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, Ashish Sabharwal. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
2021.
Introduction
Modern AI models often struggle with "multi-hop" reasoning—tasks that require combining multiple pieces of information or performing discrete operations (like math) to reach an answer. While "black-box" neural networks can be trained to solve these, they are notoriously uninterpretable; we don't know how they arrived at an answer.
Previous attempts to build modular or decomposition-based systems faced two major hurdles: the Annotation Bottleneck, requiring expensive human-annotated "blueprints" for every complex question, and the Language Gap, where a decomposition model might generate a sub-question that makes sense to a human but is phrased in a way the specific sub-model (e.g., a SQuAD-trained QA model) doesn't understand.
The authors propose Text Modular Networks (TMNs), a framework that learns to decompose tasks automatically and ensures those decompositions are "in-scope" for the models being used.
Methods
The authors introduce a specific implementation called MODULARQA. The architecture consists of two main parts:
- Sub-models ($A$): Existing "black-box" tools, such as a neural QA model (RoBERTa-Large trained on SQuAD) and a symbolic calculator.
- Next-Question Generator ($D$): A BART-based model that looks at the complex question and any previous sub-questions/answers to decide the next sub-question and the appropriate sub-model.
The "Secret Sauce": Distant Supervision
To train the generator without human-labeled decompositions, the authors use distant supervision. They identify "hints" (intermediate numbers or entities) that could lead to the final answer. A separate "sub-task question model" generates questions that would result in those hints, ensuring the generator learns to produce questions that the sub-models are actually capable of answering—effectively learning to speak the sub-model's "language."
Inference
During prediction, the system uses a best-first search to explore multiple possible decomposition chains. It uses a scoring function to pick the most plausible path, ensuring the sub-questions use words from the original query and lead to a confident final answer.
Results
The system was evaluated on two major benchmarks: DROP (numerical reasoning) and HotpotQA (multi-hop reasoning).
- Competitive Performance: On DROP, MODULARQA achieved an 87.9 F1 score, only 3.7% behind the state-of-the-art black-box model (NumNet+V2), while providing full interpretability.
- Superior Robustness: On the DROP "contrast set" (adversarial perturbations), MODULARQA outperformed the state-of-the-art black-box model by 7% F1.
- Sample Efficiency: When trained on only 20% of the available data, MODULARQA outperformed black-box models, suggesting its modular structure provides a powerful inductive bias.
- Human Preference: Human evaluators found MODULARQA’s explanations more trustworthy and understandable than prior decomposition models in approximately 70% of cases.
Conclusion
Text Modular Networks demonstrate that complex reasoning can be achieved by orchestrating existing models through natural language. By learning to decompose tasks into the specific input distribution of sub-models, MODULARQA provides faithful, human-readable explanations and superior robustness compared to traditional black-box systems. Future work includes expanding the reasoning scope to tasks like counting and sorting, and optimizing the search process for faster inference.
Summary updated