Deep learning theory has a favorite simplification: homogeneity. A function $f(\theta)$ is positively homogeneous of degree $L$ if scaling its parameters by a constant scales its output predictably: $f(c\cdot\theta) = c^L f(\theta)$.
If you remove the bias terms from a standard neural network with Rectified Linear Unit (ReLU) activations, the entire network becomes a beautifully homogeneous function. For years, this structural assumption has been the bedrock of deep learning theory. It allows researchers to write clean proofs about how neural networks optimize, why they generalize, and what kinds of solutions they prefer.
But real-world neural networks are not homogeneous. They use bias terms ($Wx + b$), residual connections ($x + f(x)$), Layer Normalization, and non-homogeneous activations like Softmax, Sigmoid, or GeLU.
Recently, a major push in deep learning theory has emerged to break free from the homogeneity constraint. By understanding exactly what happens when homogeneity is violated, researchers are bridging the gap between idealized mathematical models and the complex architectures used in production.
The Mathematical Elegance of Homogeneous Models
To understand why breaking the homogeneity assumption is difficult, we must first understand why theorists rely on it. Homogeneity introduces a predictable symmetry to the loss landscape: scaling the weights changes the magnitude of the output but preserves the directional geometry.
This symmetry yields several foundational theoretical guarantees:
- Implicit Bias and Margin Maximization: When training with gradient descent on classification tasks, network weights often grow toward infinity. For homogeneous networks, Kaifeng Lyu et al. proved in Gradient Descent Maximizes the Margin of Homogeneous Neural Networks that gradient descent implicitly maximizes a normalized geometric margin. This means the model naturally pushes decision boundaries as far from the data points as possible. Nikolaos Tsilivis et al. extended this in Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks, showing that different optimizers maximize different algorithm-dependent geometric margins, provided the network remains strictly homogeneous.
- Automatic Layer Balancing: In deep networks, keeping the weight magnitudes of different layers synchronized is crucial for stable training. Simon S. Du et al. demonstrated in Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced that for homogeneous architectures, gradient flow automatically preserves the differences between the squared norms of adjacent layers.
- Directional Stability: Because scaling doesn't alter the fundamental behavior of a homogeneous function, Matus Telgarsky showed in Directional convergence and alignment in deep learning that even as weights grow infinitely large during classification training, their directions converge to a stable state.
In short, homogeneity makes the math tractable. It allows researchers to abstract away weight magnitudes and focus entirely on directional convergence.
How Real-World Components Break the Math
When we introduce standard architectural elements like bias terms or skip connections, this mathematical elegance collapses.
The Expressivity Blind Spot of Bias-Free Networks
Why not just use bias-free homogeneous networks in practice? As it turns out, removing biases severely cripples a network's representational capacity.
Ronen Basri et al. exposed a fundamental mathematical blind spot in The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies. They proved that two-layer ReLU networks without bias terms are mathematically incapable of representing or learning odd-frequency functions; their capacity is strictly limited to even-frequency components. Adding a simple additive bias term breaks this symmetry, allowing the network to approximate the full range of frequency components in a dataset.
Furthermore, removing biases forces the network's output to be exactly zero whenever the input is zero. While the authors of Fast Axiomatic Attribution for Neural Networks leveraged this property to compute exact feature attributions efficiently, it restricts the network's ability to fit complex datasets that require non-zero intercepts.
The Optimization Landscape Distorts
Adding a bias term fundamentally alters the geometry of the loss surface. In a bias-free network, the only major critical point where gradients vanish completely is often the origin.
However, in Learning a Single Neuron with Bias Using Gradient Descent, Gal Vardi et al. showed that adding a bias term to a single ReLU neuron introduces an entire manifold of sub-optimal critical points. If a negative bias causes a neuron to output zero across the entire training distribution, the gradient stalls completely, trapping optimization in a region of zero learning.
Bias-Free Neuron Landscape:
[Origin (W=0)] ---> Smooth, directional paths to Global Minimum
Neuron with Bias Landscape:
[Sub-optimal Manifold (Output=0)] ---> [Gradient Stagnation Region] ---> Stalls before Global Minimum
This structural shift means that exact analytical formulas for gradients over an entire data population—such as the closed-form population gradient derived by Yuandong Tian in An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis—become incredibly complex or impossible to compute once non-zero biases shift the activation boundaries away from the origin.
Analyzing Non-Homogeneous Models
To close the gap between theory and practice, researchers have begun developing new mathematical frameworks that do not rely on strict homogeneity.
Quasi-Homogeneity and Near-Homogeneity
Instead of abandoning homogeneity entirely, recent work generalizes the concept. Daniel Kunin et al. introduced the framework of "quasi-homogeneous" neural networks in The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks. This framework models how gradient descent regularizes complex architectures containing both biases and skip connections, revealing that they exhibit an asymmetric maximum margin bias.
Building on this, Yuhang Cai et al. tackled a long-standing open problem in Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks. By defining a novel "near-homogeneity" condition, the authors proved that gradient descent still maximizes a normalized margin and achieves directional convergence for networks with residual connections and non-homogeneous activations. However, they noted that current proofs still require a homogeneity degree of at least one, meaning that scale-invariant or strictly bounded components like Softmax attention and Layer Normalization remain an open theoretical frontier.
Provably Efficient Non-Homogeneous Learning
Other researchers are bypassing gradient descent analysis entirely to design algorithms that can handle non-homogeneous structures natively. In Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activations, Pranjal Awasthi et al. developed the first polynomial-time, sample-efficient algorithm to learn depth-2 ReLU networks with non-zero bias terms. By utilizing Hermite expansions and robust tensor decomposition, they bypassed the gradient stagnation regions that plague standard optimization in non-homogeneous settings.
The Path Forward
The transition from homogeneous to non-homogeneous model analysis represents a maturation of deep learning theory. As surveyed by Gal Vardi in On the Implicit Bias in Deep-Learning Algorithms, while the implicit biases of homogeneous networks are increasingly well-understood, modern architectures like Transformers and ResNets routinely violate these assumptions.
By developing tools like near-homogeneity conditions and advanced geometric frameworks, theorists are finally illuminating the mathematical inner workings of the complex, non-homogeneous models that drive practical artificial intelligence.
Key Foundational Work
The Learning Dynamcis of a Universal Approximator (1996) — contributed 1 concept elements to this direction.
An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis (2017) — contributed 1 concept elements to this direction.