Abstract: In the context of learning to map an input $I$ to a function $h_I:\mathcal{X}\to \mathbb{R}$, two alternative methods are compared: (i) an embedding-based method, which learns a fixed function in which $I$ is encoded as a conditioning signal $e(I)$ and the learned function takes the form $h_I(x) = q(x,e(I))$, and (ii) hypernetworks, in which the weights $\theta_I$ of the function $h_I(x) = g(x;\theta_I)$ are given by a hypernetwork $f$ as $\theta_I=f(I)$. In this paper, we define the property of modularity as the ability to effectively learn a different function for each input instance $I$. For this purpose, we adopt an expressivity perspective of this property and extend the theory of Devore et al. 1996 and provide a lower bound on the complexity (number of trainable parameters) of neural networks as function approximators, by eliminating the requirements for the approximation method to be robust. Our results are then used to compare the complexities of $q$ and $g$, showing that under certain conditions and when letting the functions $e$ and $f$ be as large as we wish, $g$ can be smaller than $q$ by orders of magnitude. This sheds light on the modularity of hypernetworks in comparison with the embedding-based method. Besides, we show that for a structured target function, the overall number of trainable parameters in a hypernetwork is smaller by orders of magnitude than the number of trainable parameters of a standard neural network and an embedding method.
Introduction
In modern machine learning, we frequently encounter scenarios involving two distinct inputs: a primary signal $x$ (e.g., an image) and a conditioning signal $I$ (e.g., a task ID, style, or class). There are two predominant architectural paradigms for handling this:
- Embedding-based methods: The conditioning signal $I$ is mapped to a vector $e(I)$ and concatenated with $x$. A single, typically large, network $q(x, e(I))$ processes the combined input.
- Hypernetworks: A separate network $f(I)$ generates the weights $\theta_I$ for a primary network $g(x; \theta_I)$.
While hypernetworks have demonstrated state-of-the-art performance in domains like meta-learning and GANs, a rigorous theoretical explanation for their superiority was previously lacking. This research by Galanti and Wolf (NeurIPS 2020) provides that foundation, proving that hypernetworks are fundamentally more "modular" and parameter-efficient than standard embedding-based conditioning.
Theoretical Framework: Modularity and Approximation
The authors define modularity as the capacity of a model to effectively learn a distinct function for every unique input instance $I$. They analyze this through the lens of Approximation Theory, extending the work of Devore et al. (1996).
Complexity Comparison
The core of the proof lies in comparing the parameter complexity ($N$) required to approximate smooth functions to an accuracy of $\epsilon$. The authors demonstrate that:
- In an embedding-based scheme, the primary network $q$ must be complex enough to handle the combined dimensions of $x$ and $I$ ($m_1 + m_2$).
- In a hypernetwork scheme, the primary network $g$ only needs to be complex enough to handle the dimension of $x$ ($m_1$).
Because the number of parameters required to approximate a function grows exponentially with input dimensionality, embedding methods hit a "dimensionality curse" much sooner than hypernetworks. This suggests that letting one network "write" the weights of another is mathematically more efficient than simply providing the conditioning signal as additional data.
Key Insights
- Depth and Expressivity: Increasing the depth of the weight-generating network ($f$) significantly improves hypernetwork performance. In contrast, increasing the depth of the embedding network ($e$) in standard models often leads to diminishing returns or performance plateaus.
- Function Generation: A hypernetwork does not merely "shift" a function (as an embedding does); it can fundamentally alter the functional form of the primary network for every input, enabling true modularity.
- Parameter Efficiency: For structured target functions, a hypernetwork can achieve the same error as a standard network while using orders of magnitude fewer trainable parameters.
Empirical Validation
The authors validated their theory using synthetic benchmarks and self-supervised tasks, including image rotation prediction and colorization on MNIST and CIFAR10.
As shown in the experimental results, hypernetworks consistently achieve lower Mean Squared Error (MSE) than embedding methods. On the MNIST rotation task, the hypernetwork's error continued to decrease as the number of layers increased, whereas the embedding method's error remained high and relatively flat. Furthermore, the training dynamics across different datasets confirm that hypernetworks effectively "specialize" the primary network for each specific task.
Conclusion
This work establishes that hypernetworks are more than just a complex architectural choice; they are a mathematically superior way to achieve modularity in neural networks. By decoupling the complexity of the conditioning signal from the primary task, hypernetworks circumvent the dimensionality curse that plagues standard embedding-based methods. While the study focuses on expressivity, the empirical success of these models suggests that standard optimization techniques like SGD are well-suited to finding these efficient, modular solutions.
Summary updated