On the geometry of solutions and on the capacity of multi-layer neural networks with ReLU activations

Carlo Baldassi, Enrico M. Malatesta, Riccardo ZecchinaarXiv 2019 · arXiv

Abstract

The Rectified Linear Unit (ReLU) has become the standard activation function in deep learning, primarily due to its ability to mitigate the vanishing gradient problem. However, the theoretical impact of ReLU on the geometric landscape of the solution space remains less understood. In this work, we apply tools from statistical physics to analyze a two-layer tree-committee machine. We demonstrate that while ReLU networks have a finite storage capacity—contrasting with the diverging capacity of threshold-based networks—they possess a unique solution landscape characterized by dense, "flat" regions. These regions facilitate the discovery of robust models that exhibit superior performance under noise and perturbations. Our results suggest that the efficacy of ReLUs is fundamentally tied to the creation of a navigable and robust geometric landscape.

Introduction

The adoption of ReLU activations was a pivotal moment in the development of deep neural networks. While empirical success is undeniable, classical theoretical results on neural network capacity were largely derived for threshold (sign) activation functions. In such models, capacity often grows indefinitely with the number of hidden neurons. This paper investigates whether these properties hold for ReLU networks and explores the underlying geometry of their weight configurations. Specifically, we ask whether solutions are isolated points or part of larger, accessible clusters.

Theoretical Framework and Methodology

We analyze a two-layer "tree-committee machine," a simplified architecture where hidden neurons process non-overlapping subsets of input units. We consider both binary weights ($\pm 1$) and spherical weights (continuous values on a hypersphere). To map the solution landscape, we employ several techniques from the statistical mechanics of disordered systems:

  • Replica Method: To calculate the critical capacity ($\alpha_c$), the maximum ratio of training patterns to weights.
  • Franz-Parisi Entropy: To measure the distance between solutions and identify the presence of clusters.
  • Large Deviation Analysis: To locate rare regions of high solution density, corresponding to "flat minima."
  • Numerical Simulations: Using Replicated Simulated Annealing to validate analytical predictions.

The study utilizes synthetic random datasets with i.i.d. inputs $\xi \in \{-1, 1\}^N$ and random binary labels $\sigma \in \{-1, 1\}$, allowing for a fundamental study of architectural limits independent of specific dataset biases.

The Geometry of Solutions

A key finding of our analysis is the distinct nature of the ReLU solution landscape. By calculating the Franz-Parisi entropy, we can distinguish between isolated solutions and those belonging to dense clusters.

Franz-Parisi entropy as a function of distance for ReLU and sign activations.

As shown in the figure above, the entropy profile reveals the existence of clusters. While typical solutions in these networks may be isolated, large deviation analysis highlights rare, extremely dense regions. These "wide flat minima" are of significant interest as modern optimizers like Stochastic Gradient Descent (SGD) are believed to be biased toward such regions, which offer better generalization and stability.

Capacity and Robustness

Contrary to threshold-based networks where capacity diverges with the number of hidden neurons $K$, ReLU networks exhibit a finite capacity. For spherical weights, this limit saturates at approximately 2.934 patterns per weight. For binary weights, the ReLU capacity ($\alpha_c \approx 0.9039$) is slightly lower than that of sign-activation networks ($\alpha_c \approx 0.9469$).

However, the primary advantage of ReLUs lies in the quality of the solutions found within dense clusters.

Distribution of stabilities (left) and average errors vs noise (right) for ReLU and sign activations.

The stability distribution for ReLU solutions is more concentrated and centered further from the decision threshold compared to sign-activation solutions. This increased stability translates directly into superior robustness; ReLU networks can tolerate significantly higher levels of noise in both weights and input data before performance degrades. This suggests that the success of ReLUs is not merely a matter of gradient flow, but a consequence of a more favorable geometric landscape for learning.

Conclusion and Future Work

Our work provides a theoretical basis for the effectiveness of ReLU activations, emphasizing the role of solution geometry over simple gradient considerations. While the study is limited to two-layer tree-like architectures and i.i.d. data, it opens several avenues for future research. Extending these tools to deeper architectures and structured datasets will be crucial for a complete understanding of modern deep learning. Furthermore, exploring how different loss functions can further exploit these wide flat minima remains a promising direction for improving model robustness and generalization.

Summary updated

Abstract

Rectified Linear Units (ReLU) have become the main model for the neural units in current deep learning systems. This choice has been originally suggested as a way to compensate for the so called vanishing gradient problem which can undercut stochastic gradient descent (SGD) learning in networks composed of multiple layers. Here we provide analytical results on the effects of ReLUs on the capacity and on the geometrical landscape of the solution space in two-layer neural networks with either binary or real-valued weights. We study the problem of storing an extensive number of random patterns and find that, quite unexpectedly, the capacity of the network remains finite as the number of neurons in the hidden layer increases, at odds with the case of threshold units in which the capacity diverges. Possibly more important, a large deviation approach allows us to find that the geometrical landscape of the solution space has a peculiar structure: while the majority of solutions are close in distance but still isolated, there exist rare regions of solutions which are much more dense than the similar ones in the case of threshold units. These solutions are robust to perturbations of the weights and can tolerate large perturbations of the inputs. The analytical results are corroborated by numerical findings.
BibTeX
@misc{baldassi2019geometry,
  author = {Carlo Baldassi and Enrico M. Malatesta and Riccardo Zecchina},
  title = {{On the geometry of solutions and on the capacity of multi-layer neural networks with ReLU activations}},
  year = {2019},
  eprint = {1907.07578},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.07578},
}
Figure