To understand how deep neural networks learn, machine learning theorists often strip away their most defining feature: non-linear activation functions.
By analyzing deep linear networks—models where every layer computes a purely linear transformation—theorists can bypass the chaotic, non-convex landscapes of practical deep learning. On paper, a deep linear network collapses into a single matrix multiplication: $f(x) = W_L W_{L-1} \dots W_1 x = W_{\text{eff}} x$. Yet, because the weights are updated layer by layer via gradient descent, the optimization dynamics remain highly non-linear and surprisingly rich.
This simplification has yielded profound insights into implicit bias, convergence rates, and representation trajectories. However, a significant theoretical gap remains. Real-world neural networks rely on non-linearities like ReLU, GeLU, and Softmax to approximate complex functions. When we insert these non-linear boundaries, the elegant mathematical guarantees derived from linear models frequently break down.
The Tractability of Deep Linear Models
Why do theorists study linear networks in the first place? The answer lies in their mathematical tractability. They allow researchers to isolate the effects of depth and architecture from the messy geometry introduced by activation functions.
For instance, Peter Bartlett et al. proved in Gradient descent with identity initialization efficiently learns positive definite linear transformations that initializing deep linear layers at the identity matrix allows gradient descent to efficiently find optimal solutions, provided the target transformation is positive definite. This provides a clean, isolated justification for why residual connections (ResNets) stabilize training.
Linear models also help clarify how networks generalize without explicit regularization. Chulhee Yun et al. demonstrated in A unifying view on implicit bias in training linear neural networks that gradient descent naturally biases linear tensor networks toward structured solutions like max-margin or minimum-norm criteria. Similarly, in the context of compression, Mary Phuong et al. used deep linear approximations in Towards Understanding Knowledge Distillation to isolate data geometry and optimization bias as the core drivers of student network success.
Linear Network: [Input] ---> ( W1 ) ---> ( W2 ) ---> ( W3 ) ---> [Output]
\_____________________________/
Collapses to: f(x) = W_eff * x
(Smooth, tractable loss landscape)
Non-Linear Network: [Input] ---> ( W1 ) -> [ReLU] -> ( W2 ) -> [ReLU] ---> [Output]
\_____________________________________/
Cannot collapse; introduces kinks,
non-convexity, and shattered gradients
Where Linearization Fails: The Activation Gap
While deep linear networks mimic the multi-layer structure of real networks, they fail to capture how non-linearities alter the topology of the loss landscape. Activation functions introduce non-convexity, sharp thresholds, and gradient-shattering effects that fundamentally change optimization trajectories.
Sharpness and Convergence Dynamics
During training, neural networks often experience a phenomenon where the loss landscape becomes increasingly "sharp" (sensitive to parameter changes) before stabilizing. Geonhui Yoo et al. explored this in Understanding Sharpness Dynamics in NN Training with a Minimalist Example. While their minimalist, single-neuron-per-layer models replicate these dynamics, the authors note that formal mathematical guarantees do not yet extend to the complex functional mappings created by non-linear activations in wider networks.
Similarly, proofs regarding weight alignment and model convergence often require linearized architectures to remain tractable. As Zheng-An Chen et al. point out in From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics, a formal theoretical bridge is still missing to explain how standard non-linearities like GeLU or Tanh interact with these structural transitions during training.
Internal Optimization and Emergent Capabilities
The gap becomes even wider when analyzing complex architectures like Transformers. In On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability, Chenyu Zheng et al. proved that single-layer linear transformers can naturally learn to implement steps of gradient descent within their forward pass (in-context learning). However, the authors emphasize that the introduction of non-linear activation functions like softmax introduces architectural complexities that current training dynamics theories cannot fully resolve.
Case Studies in the Linear-Nonlinear Divide
The tension between linear tractability and non-linear reality spans multiple subfields of machine learning, from graph representation learning to network compression.
1. Graph Neural Networks (GNNs)
GNN theory frequently strips away non-linearities to analyze convergence and structural degradation. Two major failure modes of GNNs are oversmoothing (where node features become identical at high depths) and heterophily (poor performance when neighboring nodes belong to different classes).
- Yujun Yan et al. provided a unified node-level theory for these bugs in Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks, but noted that their proofs rely on simplified linear models.
- Michael Scholkemper et al. mathematically proved in Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs that normalization layers preserve feature rank. Yet, their formal characterization relies on linearized models, leaving the exact interaction between non-linear activations and normalization an open challenge.
- When analyzing optimization, Keyulu Xu et al. proved global convergence guarantees for deep GNNs with skip connections in Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth, but explicitly omitted non-linear activations like ReLU or Sigmoid to maintain tractability.
- Jian Kang similarly noted in Theoretical and Empirical Insights into the Origins of Degree Bias in Graph Neural Networks that while linearization allows for the derivation of error bounds and degree bias analysis, the precise influence of non-linearities remains mathematically opaque.
2. Network Compression and Regularization
The mathematical justifications for pruning, quantization, and regularization are similarly rooted in linear operator theory.
- Pruning & Quantization: Derivations for multi-layer pruning importance scores often rely on linear operator theory (Sejun Park et al. in Lookahead: A Far-sighted Alternative of Magnitude-based Pruning). Similarly, theoretical frameworks for weight oscillations during quantization-aware training are typically derived from linear models, leaving their stability in non-linear architectures largely unexplored (Jonathan Wenshøj et al. in Oscillations Make Neural Networks Robust to Quantization).
- Low-Rank Regularization: In Dropout as a Low-Rank Regularizer for Matrix Factorization, Jacopo Cavazza et al. established a rigorous mathematical link showing that dropout acts as a low-rank regularizer. However, this proof is restricted to linear matrix factorization, and generalizing it to deep architectures with non-linear activations remains an ongoing challenge.
3. Representational Drift
Even when a network's performance is stable, its internal representations often shift over time. Farhad Pashakhanloo demonstrated in Contribution of task-irrelevant stimuli to drift of neural representations that task-irrelevant stimuli drive this drift. However, the author highlights that current mathematical frameworks for tracking these representation dynamics focus primarily on linear systems, leaving a theoretical gap for non-linear architectures.
Bridging the Gap
To move beyond linear approximations, researchers are developing new mathematical tools capable of ingestion non-linear dynamics directly.
One promising direction involves borrowing frameworks from physics and control theory. For example, Ouns El Harzli et al. extended renormalization theory from statistical mechanics in Bayesian Treatment of the Spectrum of the Empirical Kernel in (Sub)Linear-Width Neural Networks to predict how common activation functions like ReLU influence generalization in large-scale Bayesian models. Others, such as Arnu Pretorius et al. in Learning Dynamics of Linear Denoising Autoencoders, suggest that applying advanced tools like non-linear random matrix theory will be essential to bridge the gap between linear theory and the practical realities of deep learning.
Ultimately, the goal of this research direction is not to abandon linear models, but to understand exactly where they break—allowing us to build a more rigorous, non-linear theory of deep learning.