From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport

Quentin Bouniot, Ievgen Redko, Anton Mallasto, Charlotte Laclau, Oliver Struckmeier, Karol Arndt, Markus Heinonen, Ville Kyrki, Samuel Kaski

TL;DR

This paper introduces the "non-linearity signature," a mathematically rigorous metric based on optimal transport theory to quantify the degree of non-linearity across a deep neural network's layers. By tracing a decade of vision architectures from AlexNet to Transformers, the authors reveal a fascinating historical pivot: while traditional CNNs evolved to minimize and streamline non-linearity, Vision Transformers (ViTs) completely reversed this trend, leveraging massive non-linearity to achieve state-of-the-art performance.

What Problem Does This Paper Address?

Why do certain neural network architectures vastly outperform others, even when they have a similar number of parameters, depths, or widths? While deep learning researchers intuitively know that a model's "expressive power" comes from its non-linearities, the community has lacked a unified, theoretically sound way to actually measure this property.

Traditional metrics like depth and width fail to explain the massive performance gaps between different architectural families (e.g., ResNets vs. Transformers). To bridge this gap, the authors tackle a fundamental question: How can we quantitatively measure the non-linearity of a deep neural network, and what does it tell us about how these models learn?

Methods

The authors introduce the affinity score ($\rho_{aff}$), a novel metric rooted in Optimal Transport (OT) theory.

For any given activation function within a network, let $X$ be the pre-activation input and $Y = \sigma(X)$ be the post-activation output. The method works in three main steps:

  1. Affine OT Alignment: It computes the best possible affine (linear + bias) transformation ($T_{aff}$) to map $X$ to $Y$.
  2. Mismatch Measurement: It calculates the 2-Wasserstein distance between this linear approximation and the actual non-linear output.
  3. Normalization: The distance is normalized by the variance of the output data, bounding the score between 0 and 1.

A score of 1 means the activation behaves perfectly linearly on the input data, while a score of 0 means it is maximally non-linear. By computing this score for every activation layer across a network, they construct a non-linearity signature—a vector that maps the progression of non-linearity throughout the model's depth.

Figure 1. Illustration of how the non-linearity of a given neural network is measured. (Top) The non-linearity signature of a DNN is a collection of affinity scores calculated for each activation function spread across its hidden layers. (Bottom) The affinity score is calculated based on 3 main steps. First, given an input (grey) and an output (red) of an activation function (left), we estimate the best affine OT fit Taff(X) (green) transporting the input to the output (middle-left). Second, we measure the mismatch between the two by summing the transportation costs (middle-right) to obtain the Wasserstein distance W2(TaffX, Y ). Finally, this distance is normalized with the magnitudes of variance (arrows in the rightmost plot) of the output data based on its covariance matrix.

Datasets

The non-linearity signatures were primarily evaluated using the ImageNet-1K validation set (50,000 images) across 36 landmark pretrained vision models. To evaluate robustness against different data distributions and training paradigms, the authors also tested the models on CIFAR10, CIFAR100, and a purely Random synthetic dataset.

Key Insights and Ideas

  • The Great Transformer U-Turn: Historically, from AlexNet (2012) to EfficientNet (2019), the dominant trend in computer vision design was actually to decrease average non-linearity or tightly constrain its spread. Vision Transformers (ViTs), introduced in 2020, completely shattered this paradigm, displaying an unprecedentedly high level of non-linearity.
  • Residual Connections act as "Linearizers": The non-linearity signature perfectly captures the mathematical role of skip connections. In architectures like ResNet, residual connections introduce a distinct bimodal distribution of affinity scores, forcing specific post-residual layers to become highly linear (scores near 1.0). This provides principled evidence that skip connections successfully preserve identity mappings.
  • Non-Linearity is Dynamic: An activation function's non-linearity is not static. As shown in Figure 2, a piecewise linear function like ReLU can be highly linear or highly non-linear depending entirely on the dynamic range and distribution of the input data it receives at that specific layer.
Non-linearity behavior of different activation functions and polynomials.

Key Results

  • Predicting Accuracy: Simple statistics extracted from a model's non-linearity signature correlate remarkably well with its final classification performance. For example, the non-linearity metrics achieved an $R^2$ score of 0.974 for ViTs and 0.814 for Residual networks when predicting ImageNet accuracy—frequently outperforming traditional baseline predictors like depth or GFLOPS.
  • Uniqueness of the Metric: The authors compared the non-linearity signature against other common layer-wise metrics (such as Centered Kernel Alignment (CKA), sparsity, and entropy). The affinity score showed low, inconsistent correlations with these metrics across different architectures, proving that it captures a fundamentally unique structural property of DNNs.

Limitations and Future Work

While highly effective, the current framework focuses its measurement entirely on activation functions as the source of non-linearity. It does not explicitly isolate the non-linearity introduced by other complex operations, such as the softmax step inside the self-attention mechanism of Transformers (though its downstream effects on the activations are captured).

Because the affinity score is fully differentiable, the authors suggest an exciting avenue for future work: using the non-linearity signature as a regularization term during training. This would allow engineers to explicitly control and shape a network's inductive bias, potentially leading to more sample-efficient or robust architectures.

Summary updated

Abstract

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications. The code for our work is available at https://github.com/qbouniot/AffScoreDeep
BibTeX
@inproceedings{bouniot2025from,
  author = {Quentin Bouniot and Ievgen Redko and Anton Mallasto and Charlotte Laclau and Oliver Struckmeier and Karol Arndt and Markus Heinonen and Ville Kyrki and Samuel Kaski},
  title = {{From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02351},
}
Figure