One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks

Atish Agarwala, Abhimanyu Das, Brendan Juba, Rina Panigrahy, Vatsal Sharan, Xin Wang, Qiuyi Zhang

Introduction

In both neuroscience and classical machine learning, modularity has long been considered a prerequisite for complex intelligence. The human brain relies on specialized regions for distinct functions like vision and language, and early machine learning research suggested that training a single network on unrelated tasks would lead to catastrophic "interference"—where learning one concept actively degrades the performance of another.

However, the rise of modern "foundation models" (such as GPT-4 or T5) challenges this assumption. These models successfully handle hundreds of diverse tasks within a single, monolithic architecture. This paper seeks to bridge the gap between empirical success and theoretical understanding, asking a fundamental question: Is modularity a strict requirement for learnability, or is it merely an optimization for computational efficiency and interpretability?

Methods and the Task Coding Framework

The authors approach this problem by treating neural networks as "universal circuit interpreters." Leveraging the Neural Tangent Kernel (NTK) framework, they prove that wide, two-layer ReLU networks can learn any multivariate analytic function using standard stochastic gradient descent (SGD).

To build this theory, they develop a "calculus" for learnability. They mathematically demonstrate that if two individual functions are learnable, their sum, product, ratio, and composition are also learnable with quantifiable sample complexity.

Crucially, the authors formalize how multiple tasks are presented to a single network through a Task Coding Framework. Tasks can be "coded" in several ways:

  • Clusters: Inputs for different tasks occupy distinct regions of the input space.
  • Decision Trees: A "task code" attribute guides the network through branching logic to reach the correct sub-task.
  • Programming Constructs: Complex, structured logic, such as SQL-style queries or Newtonian physics equations.
Examples of diverse task formulations, including N-body physics, decision trees, and SQL-style relational queries.

Key Insights and Results

The authors validated their theoretical bounds using a mix of synthetic data (Gaussian clusters, random decision trees) and structured real-world data (the Penn World Table for economic data, and simulated N-Body physics).

Overstating Interference

Theoretically, as long as task codes are "well-separated"—either spatially or via clear logical attributes—a monolithic network can learn to switch between tasks without significant interference. For $k$ tasks separated into clusters, the total data required scales roughly as the sum of the data needed for each individual task. The network does not inherently "get confused" by the sheer volume of tasks.

The Sample Complexity Penalty

While multiple tasks can be learned simultaneously, doing so incurs a "coding penalty." The authors introduce a complexity measure based on the $L_1$ norm of a function's Taylor coefficients (the "Tilde" function, $\tilde{g}$), which predicts exactly how much data a network needs to learn a specific program. For instance, learning a decision tree of depth $h$ incurs a sample complexity that grows quasipolynomially ($d^h$), matching known lower bounds for learning parities.

Neural Networks as SQL Engines and Physicists

Two of the most striking empirical results demonstrate the flexibility of monolithic networks:

  1. SQL-Style Logic: A standard two-layer network can learn to execute complex aggregation queries (e.g., SELECT AVG(income) ... WHERE zip_code = X) over a fixed database simply by observing input-output examples. It effectively learns both the logic of the query and the underlying data simultaneously, achieving R-squared values over 0.9.
  2. Learning Physical Laws: The authors proved and demonstrated that networks can learn gravitational force laws with singularities (like $1/r^2$). Interestingly, ReLU networks, which possess "slowly decaying" kernels, proved theoretically and practically superior to Gaussian kernels for learning these physical laws.

Limitations and Future Work

While the paper provides strong theoretical backing for monolithic architectures, the formal proofs are currently limited to two-layer networks. Extending these bounds to deep "monolithic" learning remains an open challenge.

Furthermore, the authors acknowledge a trade-off between learnability and efficiency. Even if a monolithic network can learn everything, modular architectures might still be preferable for reducing energy consumption (by only activating relevant sub-networks) and improving human interpretability. Finally, this work lays the foundation for exploring "programmatic reuse"—investigating whether a network that has learned a library of basic tasks can efficiently recombine them to solve novel, complex problems.

Summary updated

Abstract

Can deep learning solve multiple, very different tasks simultaneously? We investigate how the representations of the underlying tasks affect the ability of a single neural network to learn them jointly. We present theoretical and empirical findings that a single neural network is capable of simultaneously learning multiple tasks from a combined data set, for a variety of methods for representing tasks---for example, when the distinct tasks are encoded by well-separated clusters or decision trees over some task-code attributes. Indeed, more strongly, we present a novel analysis that shows that families of simple programming-like constructs for the codes encoding the tasks are learnable by two-layer neural networks with standard training. We study more generally how the complexity of learning such combined tasks grows with the complexity of the task codes; we find that learning many tasks can be provably hard, even though the individual tasks are easy to learn. We provide empirical support for the usefulness of the learning bounds by training networks on clusters, decision trees, and SQL-style aggregation.
BibTeX
@inproceedings{agarwala2021one,
  author = {Atish Agarwala and Abhimanyu Das and Brendan Juba and Rina Panigrahy and Vatsal Sharan and Xin Wang and Qiuyi Zhang},
  title = {{One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
}
Figure