From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport

Quentin Bouniot, Ievgen Redko, Anton Mallasto, Charlotte Laclau, Oliver Struckmeier, Karol Arndt, Markus Heinonen, Ville Kyrki, Samuel Kaski

Introduction

While the success of deep neural networks (DNNs) is often attributed to their depth or width, these metrics frequently fail to explain performance disparities between architectures of similar scale. The fundamental "secret sauce" of DNNs is their non-linearity; without it, any multi-layer network collapses into a single linear transformation. Despite its importance, a mathematically rigorous method to quantify the degree of non-linearity employed by specific architectures has been lacking. This research introduces the Affinity Score ($\rho_{aff}$), a metric grounded in Optimal Transport (OT) theory, to measure and analyze the non-linear behavior of landmark models.

Methodology: The Affinity Score

The proposed Affinity Score ($\rho_{aff}$) quantifies how well a layer's transformation can be approximated by an affine (linear plus translation) map. The process involves:

  1. Optimal Transport (OT): Utilizing OT theory to identify the "best-fitting" affine transformation $T_{aff}$ that maps a layer's input distribution to its output distribution.
  2. Quantification: The score compares the actual output of an activation function to this optimal affine approximation. A score of 1.0 indicates a perfectly linear layer, while a score approaching 0.0 signifies maximal non-linearity.
  3. Non-linearity Signatures: By computing $\rho_{aff}$ for every activation layer, researchers generate a "non-linearity signature"—a structural fingerprint unique to each architecture.
Overview of the Affinity Score calculation and the generation of a non-linearity signature from a neural network.

This approach is computationally efficient as it relies on a closed-form solution using the means and covariances of signals, avoiding expensive iterative optimization.

By analyzing a "hall of fame" of architectures—from AlexNet to modern Transformers—using the ImageNet-1K validation set, several key insights emerged:

The Historical "U-Turn"

From 2012 to 2019, the evolution of state-of-the-art vision models trended toward increasing linearity. For instance, EfficientNet was found to be significantly more linear than AlexNet. This trend was abruptly reversed by Vision Transformers (ViTs), which emerged as the most non-linear architectures in the study.

The Role of Residual Connections

The Affinity Score provides a principled explanation for the success of ResNets. These models exhibit a bimodal distribution in their signatures: certain layers become highly linear to facilitate identity mappings, while others maintain high non-linearity to perform complex feature extraction.

Input-Dependent Non-linearity

A common misconception is that simple piecewise functions like ReLU are inherently "simple." However, the research demonstrates that the non-linearity of a ReLU layer is highly dependent on the range of its input data. As the input distribution shifts, the effective non-linearity of the activation changes.

The Affinity Score ($\rho_{aff}$) for a ReLU activation varies significantly depending on the distribution and range of the input data.

Key Results and Predictive Power

The Affinity Score captures a unique dimension of model behavior, showing little correlation with traditional metrics like CKA, sparsity, or entropy. Furthermore, these signatures serve as effective proxies for performance. For "Post-ViT" models, the median non-linearity score achieved an $R^2$ of 0.818 with ImageNet accuracy, significantly outperforming total depth as a predictor.

Architectural families can also be grouped effectively using Dynamic Time Warping (DTW) on their signatures, confirming that the metric captures the underlying "design philosophy" of different networks.

Conclusion and Future Directions

The introduction of the Affinity Score provides a powerful tool for understanding the internal dynamics of DNNs. While currently focused on computer vision, future applications could extend to Large Language Models (LLMs) and Reinforcement Learning. Additionally, the score could potentially be used as a regularization term during training to actively shape a model's non-linear characteristics, or as a diagnostic tool for detecting shifts in data distribution.

Summary updated

Abstract

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, particularly emphasizing the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications.
BibTeX
@inproceedings{bouniot2025from,
  author = {Quentin Bouniot and Ievgen Redko and Anton Mallasto and Charlotte Laclau and Oliver Struckmeier and Karol Arndt and Markus Heinonen and Ville Kyrki and Samuel Kaski},
  title = {{From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02351},
}
Figure