Introduction
While the success of deep neural networks (DNNs) is often attributed to their depth or width, these metrics frequently fail to explain performance disparities between architectures of similar scale. The fundamental "secret sauce" of DNNs is their non-linearity; without it, any multi-layer network collapses into a single linear transformation. Despite its importance, a mathematically rigorous method to quantify the degree of non-linearity employed by specific architectures has been lacking. This research introduces the Affinity Score ($\rho_{aff}$), a metric grounded in Optimal Transport (OT) theory, to measure and analyze the non-linear behavior of landmark models.
Methodology: The Affinity Score
The proposed Affinity Score ($\rho_{aff}$) quantifies how well a layer's transformation can be approximated by an affine (linear plus translation) map. The process involves:
- Optimal Transport (OT): Utilizing OT theory to identify the "best-fitting" affine transformation $T_{aff}$ that maps a layer's input distribution to its output distribution.
- Quantification: The score compares the actual output of an activation function to this optimal affine approximation. A score of 1.0 indicates a perfectly linear layer, while a score approaching 0.0 signifies maximal non-linearity.
- Non-linearity Signatures: By computing $\rho_{aff}$ for every activation layer, researchers generate a "non-linearity signature"—a structural fingerprint unique to each architecture.
This approach is computationally efficient as it relies on a closed-form solution using the means and covariances of signals, avoiding expensive iterative optimization.
Architectural Trends and Insights
By analyzing a "hall of fame" of architectures—from AlexNet to modern Transformers—using the ImageNet-1K validation set, several key insights emerged:
The Historical "U-Turn"
From 2012 to 2019, the evolution of state-of-the-art vision models trended toward increasing linearity. For instance, EfficientNet was found to be significantly more linear than AlexNet. This trend was abruptly reversed by Vision Transformers (ViTs), which emerged as the most non-linear architectures in the study.
The Role of Residual Connections
The Affinity Score provides a principled explanation for the success of ResNets. These models exhibit a bimodal distribution in their signatures: certain layers become highly linear to facilitate identity mappings, while others maintain high non-linearity to perform complex feature extraction.
Input-Dependent Non-linearity
A common misconception is that simple piecewise functions like ReLU are inherently "simple." However, the research demonstrates that the non-linearity of a ReLU layer is highly dependent on the range of its input data. As the input distribution shifts, the effective non-linearity of the activation changes.
Key Results and Predictive Power
The Affinity Score captures a unique dimension of model behavior, showing little correlation with traditional metrics like CKA, sparsity, or entropy. Furthermore, these signatures serve as effective proxies for performance. For "Post-ViT" models, the median non-linearity score achieved an $R^2$ of 0.818 with ImageNet accuracy, significantly outperforming total depth as a predictor.
Architectural families can also be grouped effectively using Dynamic Time Warping (DTW) on their signatures, confirming that the metric captures the underlying "design philosophy" of different networks.
Conclusion and Future Directions
The introduction of the Affinity Score provides a powerful tool for understanding the internal dynamics of DNNs. While currently focused on computer vision, future applications could extend to Large Language Models (LLMs) and Reinforcement Learning. Additionally, the score could potentially be used as a regularization term during training to actively shape a model's non-linear characteristics, or as a diagnostic tool for detecting shifts in data distribution.
Summary updated