Abstract
The Rectified Linear Unit (ReLU) has become the standard activation function in deep learning, primarily due to its ability to mitigate the vanishing gradient problem. However, the theoretical impact of ReLU on the geometric landscape of the solution space remains less understood. In this work, we apply tools from statistical physics to analyze a two-layer tree-committee machine. We demonstrate that while ReLU networks have a finite storage capacity—contrasting with the diverging capacity of threshold-based networks—they possess a unique solution landscape characterized by dense, "flat" regions. These regions facilitate the discovery of robust models that exhibit superior performance under noise and perturbations. Our results suggest that the efficacy of ReLUs is fundamentally tied to the creation of a navigable and robust geometric landscape.
Introduction
The adoption of ReLU activations was a pivotal moment in the development of deep neural networks. While empirical success is undeniable, classical theoretical results on neural network capacity were largely derived for threshold (sign) activation functions. In such models, capacity often grows indefinitely with the number of hidden neurons. This paper investigates whether these properties hold for ReLU networks and explores the underlying geometry of their weight configurations. Specifically, we ask whether solutions are isolated points or part of larger, accessible clusters.
Theoretical Framework and Methodology
We analyze a two-layer "tree-committee machine," a simplified architecture where hidden neurons process non-overlapping subsets of input units. We consider both binary weights ($\pm 1$) and spherical weights (continuous values on a hypersphere). To map the solution landscape, we employ several techniques from the statistical mechanics of disordered systems:
- Replica Method: To calculate the critical capacity ($\alpha_c$), the maximum ratio of training patterns to weights.
- Franz-Parisi Entropy: To measure the distance between solutions and identify the presence of clusters.
- Large Deviation Analysis: To locate rare regions of high solution density, corresponding to "flat minima."
- Numerical Simulations: Using Replicated Simulated Annealing to validate analytical predictions.
The study utilizes synthetic random datasets with i.i.d. inputs $\xi \in \{-1, 1\}^N$ and random binary labels $\sigma \in \{-1, 1\}$, allowing for a fundamental study of architectural limits independent of specific dataset biases.
The Geometry of Solutions
A key finding of our analysis is the distinct nature of the ReLU solution landscape. By calculating the Franz-Parisi entropy, we can distinguish between isolated solutions and those belonging to dense clusters.
As shown in the figure above, the entropy profile reveals the existence of clusters. While typical solutions in these networks may be isolated, large deviation analysis highlights rare, extremely dense regions. These "wide flat minima" are of significant interest as modern optimizers like Stochastic Gradient Descent (SGD) are believed to be biased toward such regions, which offer better generalization and stability.
Capacity and Robustness
Contrary to threshold-based networks where capacity diverges with the number of hidden neurons $K$, ReLU networks exhibit a finite capacity. For spherical weights, this limit saturates at approximately 2.934 patterns per weight. For binary weights, the ReLU capacity ($\alpha_c \approx 0.9039$) is slightly lower than that of sign-activation networks ($\alpha_c \approx 0.9469$).
However, the primary advantage of ReLUs lies in the quality of the solutions found within dense clusters.
The stability distribution for ReLU solutions is more concentrated and centered further from the decision threshold compared to sign-activation solutions. This increased stability translates directly into superior robustness; ReLU networks can tolerate significantly higher levels of noise in both weights and input data before performance degrades. This suggests that the success of ReLUs is not merely a matter of gradient flow, but a consequence of a more favorable geometric landscape for learning.
Conclusion and Future Work
Our work provides a theoretical basis for the effectiveness of ReLU activations, emphasizing the role of solution geometry over simple gradient considerations. While the study is limited to two-layer tree-like architectures and i.i.d. data, it opens several avenues for future research. Extending these tools to deeper architectures and structured datasets will be crucial for a complete understanding of modern deep learning. Furthermore, exploring how different loss functions can further exploit these wide flat minima remains a promising direction for improving model robustness and generalization.
Summary updated