Comparing the Parameter Complexity of Hypernetworks and the Embedding-Based Alternative

Tomer Galanti, Lior Wolf

Abstract

In the context of learning to map an input $I$ to a function $h_I:\mathcal{X}\to \mathbb{R}$, two alternative methods are compared: (i) an embedding-based method, which learns a fixed function in which $I$ is encoded as a conditioning signal $e(I)$ and the learned function takes the form $h_I(x) = q(x,e(I))$, and (ii) hypernetworks, in which the weights $\theta_I$ of the function $h_I(x) = g(x;\theta_I)$ are given by a hypernetwork $f$ as $\theta_I=f(I)$. In this paper, we define the property of modularity as the ability to effectively learn a different function for each input instance $I$. For this purpose, we adopt an expressivity perspective of this property and extend the theory of Devore et al. 1996 and provide a lower bound on the complexity (number of trainable parameters) of neural networks as function approximators, by eliminating the requirements for the approximation method to be robust. Our results are then used to compare the complexities of $q$ and $g$, showing that under certain conditions and when letting the functions $e$ and $f$ be as large as we wish, $g$ can be smaller than $q$ by orders of magnitude. This sheds light on the modularity of hypernetworks in comparison with the embedding-based method. Besides, we show that for a structured target function, the overall number of trainable parameters in a hypernetwork is smaller by orders of magnitude than the number of trainable parameters of a standard neural network and an embedding method.

Introduction

In modern machine learning, conditioning a model on auxiliary information $I$ (such as a task ID, a style vector, or a context image) to produce a specialized function $h_I(x)$ is a common requirement. This is typically achieved through two paradigms: embedding-based conditioning and hypernetworks. Embedding methods utilize a fixed primary network $q$ that accepts both the input $x$ and a learned embedding $e(I)$. Hypernetworks, conversely, use a secondary network $f(I)$ to generate the weights $\theta_I$ for a primary network $g(x)$. While hypernetworks have demonstrated empirical success in meta-learning and generative modeling, the theoretical basis for their performance over simpler embedding methods has remained under-explored. This work investigates the property of modularity—the capacity to learn distinct functions for unique inputs—from an approximation theory perspective.

Theoretical Framework

We define modularity as the ability of a primary network $g$ to maintain the same minimal complexity as a network trained specifically for a single fixed task. To analyze this, we extend the theory of $N$-width, which measures the approximation capabilities of a function class. By removing the traditional requirement for "robustness" (stability under small perturbations), we provide a more realistic lower bound on the number of trainable parameters required for neural networks.

Our complexity analysis compares the number of parameters $N_q$ (for embedding methods) and $N_g$ (for hypernetworks) needed to achieve a target error $\epsilon$. We demonstrate that for structured target functions, the total parameter count for a hypernetwork scales as $O(\epsilon^{-m_2/r} + \epsilon^{-m_1/r})$, whereas standard embedding methods require $\Omega(\epsilon^{-(m_1+m_2)/r})$. This indicates that hypernetworks can be exponentially more efficient as the required precision increases.

Experimental Results

Scaling and Complexity

We validated our theoretical findings using synthetic target functions designed to test complex interactions. As shown in Figure 1, hypernetworks exhibit superior scaling behavior. While embedding methods often plateau in performance regardless of the embedding size or the depth of the conditioning network, hypernetworks consistently reduce Mean Squared Error (MSE) as their complexity increases.

MSE comparison between hypernetworks and embedding methods across varying depths (a) and embedding sizes (b).

Real-World Benchmarks

The modularity of hypernetworks was further tested on self-supervised tasks using MNIST and CIFAR10, specifically image rotation prediction and colorization. In these tasks, the hypernetwork must adapt the primary network to the specific context of the input image.

In the colorization task, hypernetworks achieved significantly lower error rates than embedding-based alternatives. The ability of the hypernetwork to generate the entire weight set for the primary network allows for a level of specialization that concatenation-based embeddings cannot match. Figure 2 illustrates the convergence dynamics across different datasets, highlighting the consistent performance advantage of the hypernetwork approach.

MSE convergence over training epochs for MNIST (a), CIFAR10 (b), and a synthetic set (c).

Discussion and Conclusion

Our results suggest that hypernetworks are not merely an engineering heuristic but a fundamentally more efficient architecture for modular learning. By splitting the task logic (handled by $f$) from the primary processing (handled by $g$), hypernetworks allow the primary network to remain specialized and compact.

While we have proven that low-parameter hypernetworks exist for these tasks, future work should address the optimization challenges involved in training these models with standard gradient-based methods. Additionally, exploring the generalization benefits of this modularity remains a promising avenue for research.

Summary updated

Abstract

In the context of learning to map an input $I$ to a function $h_I:\mathcal{X}\to \mathbb{R}$, two alternative methods are compared: (i) an embedding-based method, which learns a fixed function in which $I$ is encoded as a conditioning signal $e(I)$ and the learned function takes the form $h_I(x) = q(x,e(I))$, and (ii) hypernetworks, in which the weights $\theta_I$ of the function $h_I(x) = g(x;\theta_I)$ are given by a hypernetwork $f$ as $\theta_I=f(I)$. In this paper, we define the property of modularity as the ability to effectively learn a different function for each input instance $I$. For this purpose, we adopt an expressivity perspective of this property and extend the theory of Devore et al. 1996 and provide a lower bound on the complexity (number of trainable parameters) of neural networks as function approximators, by eliminating the requirements for the approximation method to be robust. Our results are then used to compare the complexities of $q$ and $g$, showing that under certain conditions and when letting the functions $e$ and $f$ be as large as we wish, $g$ can be smaller than $q$ by orders of magnitude. This sheds light on the modularity of hypernetworks in comparison with the embedding-based method. Besides, we show that for a structured target function, the overall number of trainable parameters in a hypernetwork is smaller by orders of magnitude than the number of trainable parameters of a standard neural network and an embedding method.
BibTeX
@inproceedings{galanti2020comparing,
  author = {Tomer Galanti and Lior Wolf},
  title = {{Comparing the Parameter Complexity of Hypernetworks and the Embedding-Based Alternative}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=JubpC6dgUVQq},
}
Figure