Abstract
Capacity analysis has been recently introduced as a way to analyze how linear models distribute their modelling capacity across the input space. In this paper, we extend the notion of capacity allocation to the case of neural networks with non-linear layers. We show that under some hypotheses the problem is equivalent to linear capacity allocation, within some extended input space that factors in the non-linearities. We introduce the notion of layer decoupling, which quantifies the degree to which a non-linear activation decouples its outputs, and show that it plays a central role in capacity allocation through layers. In the highly non-linear limit where decoupling is total, we show that the propagation of capacity throughout the layers follows a simple markovian rule, which turns into a diffusion PDE in the limit of deep networks with residual layers. This allows us to recover some known results about deep neural networks, such as the size of the effective receptive field, or why ResNets avoid the shattering problem.
Introduction
Neural networks are often described as "black boxes" because it is difficult to interpret how they prioritize different parts of the input space. While "capacity allocation analysis" was previously developed to quantify modeling power in linear models, extending this to deep non-linear networks has been a significant challenge. This paper addresses this by proposing a framework where capacity "flows" backward through layers, governed by the properties of non-linear activations.
Theoretical Framework: Linearizing the Non-Linear
To bridge the gap between linear and non-linear analysis, the author introduces the concept of an augmented input space. By factoring non-linearities into a higher-dimensional space, the problem can be linearized. This theoretical construct allows for the derivation of tractable equations that describe how capacity at the output layer maps back to the original input data.
Layer Decoupling and Capacity Flow
A key innovation is the introduction of a layer decoupling scale ($\nu$). This metric quantifies how much an activation function (like ReLU or Sigmoid) independentizes its outputs.
- Linear layers have no decoupling ($\nu=1$).
- Pseudo-random layers represent a theoretical limit of total decoupling ($\nu=0$).
- ReLU and Leaky ReLU sit in the middle (e.g., $\nu=0.5$ for ReLU).
In the limit of total decoupling, capacity propagation follows a linear Markovian rule: $\kappa^{l-1} = D_l \cdot \kappa^l$, where $D_l$ is a column-stochastic matrix. This implies that total capacity is conserved as it moves through the network. In deep architectures, this discrete propagation evolves into a Diffusion Partial Differential Equation (PDE), allowing the use of physical modeling tools to study network behavior.
Explaining Network Architectures
The diffusion framework provides theoretical explanations for several empirical observations in deep learning:
- The Diffusive Receptive Field: The "effective receptive field"—the input area influencing a specific neuron—grows as the square root of the network depth ($\sqrt{L}$). This is a direct consequence of the diffusion process.
- ResNets and Information Shattering: In standard deep networks, capacity can become "shattered" or disorganized. Residual connections (ResNets) stabilize this flow, ensuring that the most direct path through the network maintains significant weight and preserves local structure.
Conclusion and Limitations
By linearizing non-linear networks in an augmented space and treating capacity as a diffusive quantity, this work provides a rigorous mathematical lens for understanding deep learning architectures. However, the analysis currently relies on i.i.d. assumptions for inputs and focuses primarily on piecewise linear activations. Extending this to learned weights and smooth non-linearities like Sigmoid remains an area for future research.
Summary updated