On the Modularity of Hypernetworks

Tomer Galanti, Lior Wolf

TL;DR

This paper theoretically and empirically demonstrates why hypernetworks (neural networks that generate weights for other networks) are more parameter-efficient than standard embedding-based conditioning methods. By leveraging approximation theory, the authors prove that hypernetworks possess a unique "modularity" that allows their primary networks to be orders of magnitude smaller than those in embedding methods, while achieving the same or better error rates.

The Problem: Conditioning Neural Networks

In machine learning, models often need to process a primary input $x$ conditioned on some context or task input $I$. There are two standard ways to handle this:

  1. Embedding-based methods: Embed the context $I$ into a vector, concatenate it with the primary input $x$, and pass both through a single primary neural network $q$.
  2. Hypernetworks: Use a secondary network $f$ that takes the context $I$ and generates the actual weights for a primary network $g$. The primary network $g$ then processes $x$.

Hypernetworks have achieved state-of-the-art results in meta-learning, continual learning, and generative modeling. However, it has remained unclear whether they possess a fundamental mathematical advantage over simple embedding and concatenation. This paper seeks to answer that question by rigorously comparing the expressive power and parameter complexity of the two paradigms.

Methods: An Approximation Theory Perspective

The authors tackle this problem through the lens of approximation theory. They analyze the minimal number of trainable parameters (complexity) required for a neural network to approximate a target function $y(x, I)$ from a family of smooth functions (a Sobolev space) up to a specific error margin $\epsilon$.

To do this, they extend existing mathematical lower bounds (specifically the $N$-width framework) on neural network complexity. Crucially, they remove previous restrictive assumptions that required the approximation method to be "robust," allowing for a more general and accurate lower bound on network size. They then mathematically compare the required sizes of the primary networks ($q$ vs. $g$) and the overall architectures of both methods.

Datasets and Tasks

The authors validate their theoretical findings using both synthetic and real-world datasets:

  • Synthetic Data: Target functions of the form $y(x, I) = \langle x, h(I) \rangle$ to test parameter scaling and approximation error in a controlled environment.
  • Real-world Datasets: MNIST and CIFAR10.
  • Task: A self-supervised image rotation prediction task, where the context $I$ is an image and $x$ is a rotated version of it. The model must predict the angle of rotation.

Key Insights: The Definition of Modularity

  • The Definition of Modularity: Hypernetworks are "modular" because their primary network $g$ only needs enough capacity to model a single task or function instance at a time. The heavy lifting of adapting to different tasks is handled entirely by the weight-generating network $f$.
  • The Embedding Bottleneck: In contrast, the primary network $q$ in an embedding method lacks this modularity. It must be large and complex enough to route and compute the entire family of possible functions simultaneously based on the concatenated input.
  • Mathematical Separation: The authors prove a strict mathematical separation between the two methods. To achieve an error of $\epsilon$, the complexity of the hypernetwork's primary network $g$ scales with the dimension of $x$ alone. Meanwhile, the embedding network $q$ scales with the combined dimensions of $x$ and $I$. This means $g$ can be orders of magnitude smaller than $q$.
  • Total Parameter Efficiency: Even when accounting for the large weight-generating network $f$, the overall number of parameters required by a hypernetwork to reach a specific error is significantly lower than that of an embedding method.

Key Results

  • Theoretical Bounds: The paper establishes strict mathematical bounds proving that hypernetworks require fewer parameters to approximate smooth functions compared to embedding methods.
  • Synthetic Performance: On synthetic datasets, hypernetworks consistently achieved lower Mean Squared Error (MSE) than embedding methods. As the number of layers increased, hypernetworks continued to improve, whereas embedding methods plateaued.
MSE vs. Number of Layers
  • Vision Tasks: On the CIFAR10 rotation prediction task, a hypernetwork with roughly the same number of parameters as an embedding baseline achieved significantly lower classification error. Furthermore, increasing the depth of the hypernetwork improved performance, whereas increasing the embedding dimension in the baseline method did not yield similar gains.
MSE vs. Embedding Size

Limitations and Future Work

The primary limitation acknowledged by the authors is that their theoretical results are based on expressivity (approximation theory). The proofs guarantee that a highly efficient, modular hypernetwork exists for a given problem, but they do not theoretically guarantee that standard optimization algorithms like Stochastic Gradient Descent (SGD) will actually find it.

However, the authors note that their empirical experiments strongly suggest SGD does successfully converge to these efficient modular solutions in practice. Future work could focus on formally analyzing the optimization dynamics and loss landscapes of hypernetworks to prove why SGD is able to find these optimal configurations.

Summary updated

Abstract

In the context of learning to map an input $I$ to a function $h_I:\mathcal{X}\to \mathbb{R}$, two alternative methods are compared: (i) an embedding-based method, which learns a fixed function in which $I$ is encoded as a conditioning signal $e(I)$ and the learned function takes the form $h_I(x) = q(x,e(I))$, and (ii) hypernetworks, in which the weights $\theta_I$ of the function $h_I(x) = g(x;\theta_I)$ are given by a hypernetwork $f$ as $\theta_I=f(I)$. In this paper, we define the property of modularity as the ability to effectively learn a different function for each input instance $I$. For this purpose, we adopt an expressivity perspective of this property and extend the theory of~\cite{devore} and provide a lower bound on the complexity (number of trainable parameters) of neural networks as function approximators, by eliminating the requirements for the approximation method to be robust. Our results are then used to compare the complexities of $q$ and $g$, showing that under certain conditions and when letting the functions $e$ and $f$ be as large as we wish, $g$ can be smaller than $q$ by orders of magnitude. This sheds light on the modularity of hypernetworks in comparison with the embedding-based method. Besides, we show that for a structured target function, the overall number of trainable parameters in a hypernetwork is smaller by orders of magnitude than the number of trainable parameters of a standard neural network and an embedding method.
BibTeX
@inproceedings{galanti2020modularity,
  author = {Tomer Galanti and Lior Wolf},
  title = {{On the Modularity of Hypernetworks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=JubpC6dgUVQq},
}
Figure