TL;DR
This paper introduces the "non-linearity signature," a mathematically rigorous metric based on optimal transport theory to quantify the degree of non-linearity across a deep neural network's layers. By tracing a decade of vision architectures from AlexNet to Transformers, the authors reveal a fascinating historical pivot: while traditional CNNs evolved to minimize and streamline non-linearity, Vision Transformers (ViTs) completely reversed this trend, leveraging massive non-linearity to achieve state-of-the-art performance.
What Problem Does This Paper Address?
Why do certain neural network architectures vastly outperform others, even when they have a similar number of parameters, depths, or widths? While deep learning researchers intuitively know that a model's "expressive power" comes from its non-linearities, the community has lacked a unified, theoretically sound way to actually measure this property.
Traditional metrics like depth and width fail to explain the massive performance gaps between different architectural families (e.g., ResNets vs. Transformers). To bridge this gap, the authors tackle a fundamental question: How can we quantitatively measure the non-linearity of a deep neural network, and what does it tell us about how these models learn?
Methods
The authors introduce the affinity score ($\rho_{aff}$), a novel metric rooted in Optimal Transport (OT) theory.
For any given activation function within a network, let $X$ be the pre-activation input and $Y = \sigma(X)$ be the post-activation output. The method works in three main steps:
- Affine OT Alignment: It computes the best possible affine (linear + bias) transformation ($T_{aff}$) to map $X$ to $Y$.
- Mismatch Measurement: It calculates the 2-Wasserstein distance between this linear approximation and the actual non-linear output.
- Normalization: The distance is normalized by the variance of the output data, bounding the score between 0 and 1.
A score of 1 means the activation behaves perfectly linearly on the input data, while a score of 0 means it is maximally non-linear. By computing this score for every activation layer across a network, they construct a non-linearity signature—a vector that maps the progression of non-linearity throughout the model's depth.
Datasets
The non-linearity signatures were primarily evaluated using the ImageNet-1K validation set (50,000 images) across 36 landmark pretrained vision models. To evaluate robustness against different data distributions and training paradigms, the authors also tested the models on CIFAR10, CIFAR100, and a purely Random synthetic dataset.
Key Insights and Ideas
- The Great Transformer U-Turn: Historically, from AlexNet (2012) to EfficientNet (2019), the dominant trend in computer vision design was actually to decrease average non-linearity or tightly constrain its spread. Vision Transformers (ViTs), introduced in 2020, completely shattered this paradigm, displaying an unprecedentedly high level of non-linearity.
- Residual Connections act as "Linearizers": The non-linearity signature perfectly captures the mathematical role of skip connections. In architectures like ResNet, residual connections introduce a distinct bimodal distribution of affinity scores, forcing specific post-residual layers to become highly linear (scores near 1.0). This provides principled evidence that skip connections successfully preserve identity mappings.
- Non-Linearity is Dynamic: An activation function's non-linearity is not static. As shown in Figure 2, a piecewise linear function like ReLU can be highly linear or highly non-linear depending entirely on the dynamic range and distribution of the input data it receives at that specific layer.
Key Results
- Predicting Accuracy: Simple statistics extracted from a model's non-linearity signature correlate remarkably well with its final classification performance. For example, the non-linearity metrics achieved an $R^2$ score of 0.974 for ViTs and 0.814 for Residual networks when predicting ImageNet accuracy—frequently outperforming traditional baseline predictors like depth or GFLOPS.
- Uniqueness of the Metric: The authors compared the non-linearity signature against other common layer-wise metrics (such as Centered Kernel Alignment (CKA), sparsity, and entropy). The affinity score showed low, inconsistent correlations with these metrics across different architectures, proving that it captures a fundamentally unique structural property of DNNs.
Limitations and Future Work
While highly effective, the current framework focuses its measurement entirely on activation functions as the source of non-linearity. It does not explicitly isolate the non-linearity introduced by other complex operations, such as the softmax step inside the self-attention mechanism of Transformers (though its downstream effects on the activations are captured).
Because the affinity score is fully differentiable, the authors suggest an exciting avenue for future work: using the non-linearity signature as a regularization term during training. This would allow engineers to explicitly control and shape a network's inductive bias, potentially leading to more sample-efficient or robust architectures.
Summary updated