From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport

Quentin Bouniot, Ievgen Redko, Anton Mallasto, Charlotte Laclau, Oliver Struckmeier, Karol Arndt, Markus Heinonen, Ville Kyrki, Samuel Kaski

TL;DR

This paper introduces the affinity score and non-linearity signature, a theoretically grounded optimal transport framework to measure how non-linear deep neural networks are across their layers. Tracking vision models from AlexNet to modern Transformers reveals an unexpected historical arc: vision architectures steadily increased their internal linearity for a decade until Vision Transformers (ViTs) abruptly reversed the trend, proving to be the most non-linear architectures developed to date.

What Problem Does This Paper Address?

Over the past decade, deep learning architectures evolved from AlexNet to ResNets, EfficientNets, and modern Vision Transformers (ViTs). Despite having comparable depth or parameter counts, different architectures often display radically different empirical performance.

Standard theoretical measures of expressive power typically rely on coarse architectural metrics like depth and width, which fail to explain why one architecture family outclasses another. While non-linear activation functions are essential for neural networks to approximate complex functions, the machine learning community lacked a principled, mathematically sound metric to quantify the degree of non-linearity throughout a network's layers.

Methods

The authors quantify non-linearity using Optimal Transport (OT) theory:

  1. Affine Optimal Transport Fit ($T_{\text{aff}}$): For pre-activation features $X$ and post-activation values $Y = \sigma(X)$, the optimal transport map that would align their Gaussian approximations can be calculated in closed form: $$T_{\text{aff}}(x) = Ax + b$$ where $A = \Sigma(Y)^{1/2}\left(\Sigma(Y)^{1/2}\Sigma(X)\Sigma(Y)^{1/2}\right)^{-1/2}\Sigma(Y)^{1/2}$ and $b = \mu(Y) - A\mu(X)$.

  2. The Affinity Score ($\rho_{\text{aff}}$): When $\sigma$ is strictly linear, the 2-Wasserstein distance $W_2(T_{\text{aff}}X, Y)$ is 0. Using the Gelbrich bound, the authors normalize this distance into an interpretable metric bounded in $[0, 1]$: $$\rho_{\text{aff}}(X, \sigma(X)) = 1 - \frac{W_2(T_{\text{aff}}X, \sigma(X))}{\sqrt{2\operatorname{Tr}[\Sigma(\sigma(X))]}}$$ A score of 1 indicates a purely affine transformation; 0 indicates maximal non-linearity.

  3. Non-Linearity Signature: The layer-wise vector of affinity scores across every activation function in a network.

  4. Practical Computation: To handle high-dimensional spatial feature maps without prohibitive compute, the authors use spatial average pooling (which preserves the score) combined with Ledoit-Wolf shrinkage covariance estimation when sample sizes are smaller than feature dimensions.

Datasets

  • ImageNet-1K: Used as the primary benchmark evaluating 33+ landmark pre-trained vision architectures across 50,000 validation images.
  • CIFAR-10 & CIFAR-100: Used for ablation studies on training dynamics, random seed sensitivity, batch size robustness, and dataset shift.
  • Synthetic Random Tensors: Used to examine non-linearity behavior in out-of-distribution / unconstrained settings.

Key Insights and Ideas

  • Historical "U-turn" in Model Linearity: From 2012 to 2020 (AlexNet $\to$ VGG $\to$ Inception $\to$ ResNet $\to$ EfficientNet), top architectures trended toward higher linearity (higher affinity scores) and larger variance/spread in affinity scores across layers. Vision Transformers (ViTs) radically broke this paradigm, operating in an unprecedentedly high non-linearity regime (scores as low as 0.1–0.2).
  • Residual Connections as Linearizers: Residual connections explicitly create a bimodal distribution of affinity scores. Post-residual activations sit near $\rho_{\text{aff}} \approx 1.0$, empirically proving that skip connections enable learning identity functions.
  • Non-Linearity Depends on Activation Domain: A piecewise linear function like ReLU can be substantially more non-linear than smooth sigmoids or tanh depending on where the pre-activation inputs land. In fact, slight shifts in initialization bias (e.g., Glorot scale) dramatically change the non-linearity of random MLPs, refuting recent claims that ReLU has an invariant "simplicity bias."
  • ViT Patches and Attention as a "Focusing Lens": ViT patch sizes strongly shape non-linearity. The authors hypothesize that multi-head self-attention acts as a lens directing embeddings into domains where GELU activations achieve maximum non-linearity.

Key Results

  • Predicting Model Accuracy: Metrics extracted from non-linearity signatures strongly predict ImageNet top-1 accuracy, outperforming traditional metrics like FLOPs and parameter counts:
  • For ViTs, the minimum affinity score correlates with accuracy with an $R^2 = 0.974$ (vs. $R^2 = 0.842$ for FLOPs).
  • For Residual Networks, $\text{Max}/\text{Median}$ ratio yields an $R^2 = 0.814$ (vs. $R^2 = 0.666$ for FLOPs).
  • For Post-ViT architectures (Swin, ConvNeXt), median non-linearity yields $R^2 = 0.818$ (vs. $R^2 = 0.788$ for depth).
  • Uniqueness Over Existing Metrics: Non-linearity signatures do not consistently correlate across architectures with traditional metrics like linear CKA, Frobenius norm difference, activation sparsity, entropy, or linear regression $R^2$ (average Pearson correlations across architectures range between $-0.44$ and $0.33$).
  • Architecture Identification Under Different Training Objectives: Non-linearity signatures remain remarkably stable across 17 ResNet-50 variants trained via self-supervised methods (DINO, MoCo, SwAV) and supervised objectives, accurately clustering models by family via Dynamic Time Warping (DTW) distance.

Limitations and Future Work

  • Focus on Vision: The experimental evaluations are predominantly centered on computer vision architectures; language models and multi-modal foundation models remain open for broad analysis.
  • Activation-Centric View: The signature isolates activation layers; non-linearities arising from soft-attention matrix operations (e.g., Softmax) are measured through their downstream effects rather than directly as isolated transport maps.
  • Future Optimization: Because the closed-form affine transport map is differentiable, future work can directly regularize non-linearity signatures during training to inject specific inductive biases or improve generalization.

Summary updated

Abstract

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, particularly emphasizing the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications.
BibTeX
@inproceedings{bouniot2025from,
  author = {Quentin Bouniot and Ievgen Redko and Anton Mallasto and Charlotte Laclau and Oliver Struckmeier and Karol Arndt and Markus Heinonen and Ville Kyrki and Samuel Kaski},
  title = {{From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02351},
}
Figure