TL;DR
This paper introduces the affinity score and non-linearity signature, a theoretically grounded optimal transport framework to measure how non-linear deep neural networks are across their layers. Tracking vision models from AlexNet to modern Transformers reveals an unexpected historical arc: vision architectures steadily increased their internal linearity for a decade until Vision Transformers (ViTs) abruptly reversed the trend, proving to be the most non-linear architectures developed to date.
What Problem Does This Paper Address?
Over the past decade, deep learning architectures evolved from AlexNet to ResNets, EfficientNets, and modern Vision Transformers (ViTs). Despite having comparable depth or parameter counts, different architectures often display radically different empirical performance.
Standard theoretical measures of expressive power typically rely on coarse architectural metrics like depth and width, which fail to explain why one architecture family outclasses another. While non-linear activation functions are essential for neural networks to approximate complex functions, the machine learning community lacked a principled, mathematically sound metric to quantify the degree of non-linearity throughout a network's layers.
Methods
The authors quantify non-linearity using Optimal Transport (OT) theory:
-
Affine Optimal Transport Fit ($T_{\text{aff}}$): For pre-activation features $X$ and post-activation values $Y = \sigma(X)$, the optimal transport map that would align their Gaussian approximations can be calculated in closed form: $$T_{\text{aff}}(x) = Ax + b$$ where $A = \Sigma(Y)^{1/2}\left(\Sigma(Y)^{1/2}\Sigma(X)\Sigma(Y)^{1/2}\right)^{-1/2}\Sigma(Y)^{1/2}$ and $b = \mu(Y) - A\mu(X)$.
-
The Affinity Score ($\rho_{\text{aff}}$): When $\sigma$ is strictly linear, the 2-Wasserstein distance $W_2(T_{\text{aff}}X, Y)$ is 0. Using the Gelbrich bound, the authors normalize this distance into an interpretable metric bounded in $[0, 1]$: $$\rho_{\text{aff}}(X, \sigma(X)) = 1 - \frac{W_2(T_{\text{aff}}X, \sigma(X))}{\sqrt{2\operatorname{Tr}[\Sigma(\sigma(X))]}}$$ A score of 1 indicates a purely affine transformation; 0 indicates maximal non-linearity.
-
Non-Linearity Signature: The layer-wise vector of affinity scores across every activation function in a network.
- Practical Computation: To handle high-dimensional spatial feature maps without prohibitive compute, the authors use spatial average pooling (which preserves the score) combined with Ledoit-Wolf shrinkage covariance estimation when sample sizes are smaller than feature dimensions.
Datasets
- ImageNet-1K: Used as the primary benchmark evaluating 33+ landmark pre-trained vision architectures across 50,000 validation images.
- CIFAR-10 & CIFAR-100: Used for ablation studies on training dynamics, random seed sensitivity, batch size robustness, and dataset shift.
- Synthetic Random Tensors: Used to examine non-linearity behavior in out-of-distribution / unconstrained settings.
Key Insights and Ideas
- Historical "U-turn" in Model Linearity: From 2012 to 2020 (AlexNet $\to$ VGG $\to$ Inception $\to$ ResNet $\to$ EfficientNet), top architectures trended toward higher linearity (higher affinity scores) and larger variance/spread in affinity scores across layers. Vision Transformers (ViTs) radically broke this paradigm, operating in an unprecedentedly high non-linearity regime (scores as low as 0.1–0.2).
- Residual Connections as Linearizers: Residual connections explicitly create a bimodal distribution of affinity scores. Post-residual activations sit near $\rho_{\text{aff}} \approx 1.0$, empirically proving that skip connections enable learning identity functions.
- Non-Linearity Depends on Activation Domain: A piecewise linear function like ReLU can be substantially more non-linear than smooth sigmoids or tanh depending on where the pre-activation inputs land. In fact, slight shifts in initialization bias (e.g., Glorot scale) dramatically change the non-linearity of random MLPs, refuting recent claims that ReLU has an invariant "simplicity bias."
- ViT Patches and Attention as a "Focusing Lens": ViT patch sizes strongly shape non-linearity. The authors hypothesize that multi-head self-attention acts as a lens directing embeddings into domains where GELU activations achieve maximum non-linearity.
Key Results
- Predicting Model Accuracy: Metrics extracted from non-linearity signatures strongly predict ImageNet top-1 accuracy, outperforming traditional metrics like FLOPs and parameter counts:
- For ViTs, the minimum affinity score correlates with accuracy with an $R^2 = 0.974$ (vs. $R^2 = 0.842$ for FLOPs).
- For Residual Networks, $\text{Max}/\text{Median}$ ratio yields an $R^2 = 0.814$ (vs. $R^2 = 0.666$ for FLOPs).
- For Post-ViT architectures (Swin, ConvNeXt), median non-linearity yields $R^2 = 0.818$ (vs. $R^2 = 0.788$ for depth).
- Uniqueness Over Existing Metrics: Non-linearity signatures do not consistently correlate across architectures with traditional metrics like linear CKA, Frobenius norm difference, activation sparsity, entropy, or linear regression $R^2$ (average Pearson correlations across architectures range between $-0.44$ and $0.33$).
- Architecture Identification Under Different Training Objectives: Non-linearity signatures remain remarkably stable across 17 ResNet-50 variants trained via self-supervised methods (DINO, MoCo, SwAV) and supervised objectives, accurately clustering models by family via Dynamic Time Warping (DTW) distance.
Limitations and Future Work
- Focus on Vision: The experimental evaluations are predominantly centered on computer vision architectures; language models and multi-modal foundation models remain open for broad analysis.
- Activation-Centric View: The signature isolates activation layers; non-linearities arising from soft-attention matrix operations (e.g., Softmax) are measured through their downstream effects rather than directly as isolated transport maps.
- Future Optimization: Because the closed-form affine transport map is differentiable, future work can directly regularize non-linearity signatures during training to inject specific inductive biases or improve generalization.
Summary updated