Understanding deep neural networks through the lens of their non-linearity

Quentin Bouniot, Ievgen Redko, Anton Mallasto, Charlotte Laclau, Karol Arndt, Oliver Struckmeier, Markus Heinonen, Ville Kyrki, Samuel Kaski

Abstract

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications. The code for our work is available at https://github.com/qbouniot/AffScoreDeep

Introduction

Despite the massive success of deep learning, explaining why certain architectures outperform others remains a challenge. Traditional metrics like depth and width are often poor predictors of performance. For instance, a Transformer and AlexNet might have similar depths, yet their performance gap is cavernous. This research argues that the missing piece is a precise measurement of non-linearity. Since activation functions (e.g., ReLU, GeLU) are the primary source of non-linearity, quantifying how "non-linear" they are during a forward pass on real data can reveal a model's intrinsic capacity.

Methodology: The Affinity Score

The authors introduce the Affinity Score ($\rho_{aff}$), grounded in Optimal Transport (OT) principles. The score quantifies the non-linearity of an activation function by comparing its actual output to the "best possible" linear (affine) approximation.

  1. Optimal Transport Mapping: The 2-Wasserstein distance measures the mismatch between the actual output and the output of an optimal affine map ($T_{aff}$).
  2. Normalization: The score is normalized between 0 and 1. A score of 1 indicates perfectly linear behavior, while 0 represents maximal non-linearity.
  3. Non-linearity Signature: By calculating $\rho_{aff}$ for every activation layer, a "signature" or vector is created, representing the model's unique non-linear fingerprint.
Overview of the non-linearity signature extraction pipeline, showing the transition from input image to a layer-wise affinity score vector.

The researchers derived a closed-form solution for this score using normal approximations of the data, making it computationally efficient even for large-scale models like Vision Transformers (ViT).

Experimental Setup

The study primarily utilized ImageNet-1K to extract signatures from landmark models. To ensure robustness, evaluations were extended to:

  • CIFAR-10/100: To assess stability across data distributions.
  • Random Tensors: To observe behavior with unorganized input.
  • Self-supervised models: Testing 17 versions of ResNet-50 trained with paradigms like MoCo, DINO, and SwAV.

Key Insights

The affinity score reveals fundamental properties of model design and function.

  • Architecture Fingerprinting: Signatures are unique enough to identify model families (e.g., VGG-style) regardless of the training paradigm.
  • Complexity vs. Weights: The non-linearity of a ReLU is not just a property of the function itself but is heavily influenced by the range of inputs it receives, which is determined by the network's weights.
  • Activation Comparisons: Different activation functions exhibit varying baseline non-linearity levels.
Comparison of affinity scores across different input ranges for ReLU (A), different activation functions (B), and varying polynomial orders (C).

Results and Discussion

  • The "U-Turn" in Evolution: From AlexNet (2012) to EfficientNet (2019), computer vision models trended toward increasing linearity, likely making information flow easier. However, Vision Transformers (2020) reversed this, becoming the most non-linear models measured to date.
  • Residual Connections: ResNets exhibit "bimodal" signatures—some layers remain highly non-linear while others become almost perfectly linear, confirming that residual connections effectively facilitate identity mappings.
  • Predictive Power: Statistics derived from these signatures often correlate better with ImageNet accuracy than traditional metrics like GFLOPS or parameter counts.

Conclusion

The non-linearity signature provides a theoretically sound framework for understanding DNN dynamics. While currently focused on computer vision, future work could extend this analysis to Large Language Models (LLMs) or use the differentiable affinity score as a regularization term during training to explicitly control model non-linearity.

Summary updated

Abstract

In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications. The code for our work is available at https://github.com/qbouniot/AffScoreDeep
BibTeX
@inproceedings{bouniot2025understanding,
  author = {Quentin Bouniot and Ievgen Redko and Anton Mallasto and Charlotte Laclau and Karol Arndt and Oliver Struckmeier and Markus Heinonen and Ville Kyrki and Samuel Kaski},
  title = {{Understanding deep neural networks through the lens of their non-linearity}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02351},
}
Figure