Abstract
In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task.
The Challenge of Measuring Model Capacity
For over a decade, AI performance has climbed steadily, from the simple convolutional layers of AlexNet to the complex attention mechanisms of Transformers. However, explaining why one architecture outperforms another remains difficult. Standard metrics like "depth" (number of layers) or "width" (number of neurons) are often too coarse; two models with identical depth can have vastly different capabilities.
The authors argue that the missing piece is a precise measurement of non-linearity. Since non-linearity is what allows a network to learn complex, non-obvious patterns, measuring it should reveal the "intrinsic capacity" of a model. Until now, there was no mathematically sound way to measure this across an entire network.
Methodology: The Non-linearity Signature
The authors propose the Affinity Score ($\rho_{aff}$), which quantifies how much an activation function (like ReLU or GeLU) deviates from a simple linear (affine) transformation.
The process involves:
- Affine Optimal Transport (OT): Using the 2-Wasserstein distance to compare the actual output of an activation layer to the "best-fit" linear approximation.
- Closed-Form Solution: Deriving a computationally efficient closed-form solution for the "best" linear map using the mean and covariance of the data.
- The Signature: Calculating this score for every activation layer in a network to create a "signature"—a vector representing the non-linearity profile of the entire architecture.
- Normalization: The score is scaled from 0 (maximally non-linear) to 1 (perfectly linear).
Key Insights
The Linearity "U-Turn"
Historically, from AlexNet (2012) to EfficientNet (2019), the trend was toward increasing linearity. Researchers were making models easier to train by making them more linear. Vision Transformers (ViTs) in 2020 completely broke this trend, leveraging much higher levels of non-linearity.
ReLU is Context-Dependent
A crucial finding is that a ReLU function isn't "just" a ReLU. Its non-linearity depends entirely on the input data distribution.
If the data is mostly positive, ReLU acts like a linear identity function ($\rho_{aff} \approx 1$); if it straddles zero, it becomes highly non-linear ($\rho_{aff}$ decreases).
Architecture as Identity
The non-linearity signature is so characteristic of an architecture that it can be used to cluster models (e.g., grouping all ResNets together) based solely on their signature, regardless of whether they were trained via supervised or self-supervised methods.
Key Results
- Accuracy Prediction: For specific model families like ResNets and Transformers, statistics derived from the non-linearity signature correlate better with ImageNet accuracy ($R^2$ up to 0.97) than traditional metrics like depth or GFLOPS.
- Stability: The signature is remarkably stable across different batch sizes or random initialization seeds, suggesting it captures an inherent property of the architecture's design.
- ViT Uniqueness: Using Dynamic Time Warping (DTW) to compare signatures, the authors showed that Vision Transformers are mathematically "dissimilar" to almost all previous convolutional architectures.
Conclusion and Future Work
While the current work focuses on Computer Vision using ImageNet-1K and CIFAR datasets, the underlying theory is general. Future work could extend this to Natural Language Processing and Large Language Models. Furthermore, the authors suggest that this score could eventually be used as a regularizer during training, allowing researchers to explicitly control the amount of non-linearity in a model.
The code for this work is available at https://github.com/qbouniot/AffScoreDeep
Summary updated