Abstract
Designing neural network architectures is a task that lies somewhere between science and art. For a given task, some architectures are eventually preferred over others, based on a mix of intuition, experience, experimentation and luck. For many tasks, the final word is attributed to the loss function, while for some others a further perceptual evaluation is necessary to assess and compare performance across models. In this paper, we introduce the concept of capacity allocation analysis, with the aim of shedding some light on what network architectures focus their modelling capacity on, when used on a given task. We focus more particularly on spatial capacity allocation, which analyzes a posteriori the effective number of parameters that a given model has allocated for modelling dependencies on a given point or region in the input space, in linear settings. We use this framework to perform a quantitative comparison between some classical architectures on various synthetic tasks. Finally, we consider how capacity allocation might translate in non-linear settings.
Introduction
Designing neural network architectures often feels like a blend of high-level engineering and "dark magic." Practitioners frequently rely on intuition and trial-and-error, using the final loss value as the primary metric of success. However, the loss function is a blunt instrument; while it indicates that a model is performing better, it rarely explains why. In his paper, Jonathan Donier introduces capacity allocation analysis, a framework designed to peek under the hood by treating a network’s parameters as a finite "budget." This approach quantifies exactly where a model distributes its modeling power across the input space.
The Framework: Capacity Allocation Analysis
The core of this analysis is the concept of Capacity (κ). To make the mathematics tractable, the framework focuses on linear or linearized versions of neural networks. In a linear system, each independent parameter acts as a "constraint" on the relationship between input and output. The total number of these effective parameters constitutes the model's total capacity.
A central component is Spatial Capacity Allocation, which measures how much of the total capacity is dedicated to specific regions or "lags" in the input space. This allows researchers to visualize the "architectural signature" of a model.
As shown in the figure above, different architectures exhibit distinct allocation patterns:
- Myopic allocation: Concentrates capacity on the most recent inputs (lag 0), rapidly decaying for more distant inputs.
- Fat-tailed allocation: Spreads capacity more broadly, allowing the model to capture longer-range dependencies.
- Uniform allocation: Distributes capacity equally across the entire input window.
Key Insights: The Zero-Sum Game of Parameters
The framework reveals that capacity is essentially a zero-sum game. Because a model has a fixed budget of effective parameters, an architecture that spends heavily on recent inputs physically cannot model the distant past effectively.
Further analysis of hierarchical models, such as Wavenet-style dilated convolutions, shows they are "greedy." They prioritize saturating capacity for short-term dependencies first. Only once the short-term modeling is "full" (capacity ≈ 1) does the budget shift toward longer-range lags. This provides a theoretical explanation for why hierarchical models are so effective: lower layers handle short-term structure, while upper layers specialize in long-term dependencies. By using "conditional capacity," the author can identify redundant layers that contribute little to the model's overall representational power.
Results and Comparative Analysis
To validate this framework, the author utilized synthetic tasks, including 1D and multi-dimensional Gaussian Processes and autoregressive tasks. These controlled environments allowed for precise measurement of how capacity scales with dimensionality and how different architectures capture long-range versus short-range dependencies.
The paper establishes a mathematical link between capacity and performance: the squared error in a specific input region is statistically bounded by the "missing" capacity in that region. If an architecture fails to allocate capacity to a certain lag, it is guaranteed to exhibit error there. This framework quantifies the success of specific architectures; for instance, Wavenets outperform simple RNNs in audio tasks because their dilated structure allows them to "skip" redundant short-term information and allocate capacity across exponential time scales.
Limitations and Future Directions
The primary limitation of this framework is the reliance on linearization. While the author argues that spatial architecture (the connectivity pattern) is often more influential than specific non-linearities for capacity distribution, the extent to which these insights hold for highly non-linear, deep ReLU networks remains an open question.
Future work aims to generalize the theory to fully non-linear settings and develop a comprehensive library of "capacity signatures" for common architectural variants. Such a library could eventually automate the pruning of redundant layers and lead to more principled, efficient neural network design.
Summary updated