From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport

Quentin Bouniot, Ievgen Redko, Anton Mallasto, Charlotte Laclau, Karol Arndt, Oliver Struckmeier, Markus Heinonen, Ville Kyrki, Samuel Kaski

Abstract

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task.

The Challenge of Measuring Model Capacity

For over a decade, AI performance has climbed steadily, from the simple convolutional layers of AlexNet to the complex attention mechanisms of Transformers. However, explaining why one architecture outperforms another remains difficult. Standard metrics like "depth" (number of layers) or "width" (number of neurons) are often too coarse; two models with identical depth can have vastly different capabilities.

The authors argue that the missing piece is a precise measurement of non-linearity. Since non-linearity is what allows a network to learn complex, non-obvious patterns, measuring it should reveal the "intrinsic capacity" of a model. Until now, there was no mathematically sound way to measure this across an entire network.

Methodology: The Non-linearity Signature

The authors propose the Affinity Score ($\rho_{aff}$), which quantifies how much an activation function (like ReLU or GeLU) deviates from a simple linear (affine) transformation.

Overview of the non-linearity signature pipeline, showing how affine optimal transport is used to compare activations to their best linear fit.

The process involves:

  1. Affine Optimal Transport (OT): Using the 2-Wasserstein distance to compare the actual output of an activation layer to the "best-fit" linear approximation.
  2. Closed-Form Solution: Deriving a computationally efficient closed-form solution for the "best" linear map using the mean and covariance of the data.
  3. The Signature: Calculating this score for every activation layer in a network to create a "signature"—a vector representing the non-linearity profile of the entire architecture.
  4. Normalization: The score is scaled from 0 (maximally non-linear) to 1 (perfectly linear).

Key Insights

The Linearity "U-Turn"

Historically, from AlexNet (2012) to EfficientNet (2019), the trend was toward increasing linearity. Researchers were making models easier to train by making them more linear. Vision Transformers (ViTs) in 2020 completely broke this trend, leveraging much higher levels of non-linearity.

ReLU is Context-Dependent

A crucial finding is that a ReLU function isn't "just" a ReLU. Its non-linearity depends entirely on the input data distribution.

Illustration of how the Affinity Score ($\rho_{aff}$) changes for a ReLU activation depending on the range of inputs.

If the data is mostly positive, ReLU acts like a linear identity function ($\rho_{aff} \approx 1$); if it straddles zero, it becomes highly non-linear ($\rho_{aff}$ decreases).

Architecture as Identity

The non-linearity signature is so characteristic of an architecture that it can be used to cluster models (e.g., grouping all ResNets together) based solely on their signature, regardless of whether they were trained via supervised or self-supervised methods.

Key Results

  • Accuracy Prediction: For specific model families like ResNets and Transformers, statistics derived from the non-linearity signature correlate better with ImageNet accuracy ($R^2$ up to 0.97) than traditional metrics like depth or GFLOPS.
  • Stability: The signature is remarkably stable across different batch sizes or random initialization seeds, suggesting it captures an inherent property of the architecture's design.
  • ViT Uniqueness: Using Dynamic Time Warping (DTW) to compare signatures, the authors showed that Vision Transformers are mathematically "dissimilar" to almost all previous convolutional architectures.

Conclusion and Future Work

While the current work focuses on Computer Vision using ImageNet-1K and CIFAR datasets, the underlying theory is general. Future work could extend this to Natural Language Processing and Large Language Models. Furthermore, the authors suggest that this score could eventually be used as a regularizer during training, allowing researchers to explicitly control the amount of non-linearity in a model.

The code for this work is available at https://github.com/qbouniot/AffScoreDeep

Summary updated

Abstract

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications. The code for our work is available at https://github.com/qbouniot/AffScoreDeep
BibTeX
@inproceedings{bouniot2025from,
  author = {Quentin Bouniot and Ievgen Redko and Anton Mallasto and Charlotte Laclau and Karol Arndt and Oliver Struckmeier and Markus Heinonen and Ville Kyrki and Samuel Kaski},
  title = {{From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02351},
}
Figure