When we observe biological learning signatures in a neural network—such as forward weights aligning with random feedback paths, or synaptic updates tracking local correlations between pre- and post-synaptic activity—it is tempting to assume these behaviors are explicitly hardcoded into the system's learning rules.
However, a recent convergence of theoretical insights suggests a more elegant explanation: these biological signatures naturally emerge from the mathematical necessity of balancing regularization constraints with optimization stability. Rather than being explicitly programmed, phenomena like Feedback Alignment and Hebbian plasticity can be understood as steady-state solutions where contractive forces (such as weight decay) and expansive forces (such as learning signals and noise) reach a geometric equilibrium.
Forcing Alignment in Feedback Systems
To understand how regularization drives structural alignment, we can look at Feedback Alignment (FA), a prominent biologically plausible alternative to backpropagation. In backpropagation, the backward pass uses the exact transpose of the forward weight matrix to transmit gradients. FA replaces this with a fixed, random matrix. Remarkably, networks trained with FA still converge because the forward weights dynamically alter themselves to "align" with the fixed random feedback weights.
How does this alignment happen? Song et al. demonstrated that this synchronization is not an inherent guarantee of the feedback signal alone; rather, it is explicitly driven by $L_2$ regularization.
Without Regularization:
Initial Random Weights ───► Dominate Trajectory ───► Poor Alignment
With L2 Regularization:
L2 Penalty ───► Suppresses Initial Random Components ───► Feedback Signal Dominates ───► Forward-Backward Alignment
By penalizing the magnitude of the weights, $L_2$ regularization suppresses the initial, arbitrary components of the forward weights. This suppression allows the persistent learning signal coming from the fixed feedback weights to exert a dominant influence on the network's structural evolution, forcing the forward weights to synchronize with the backward path.
Expanding on this mechanism, Robertson et al. uncovered a fundamental conservation law governing this process. They proved that the evolution of a neuron's incoming weights is mathematically coupled to the alignment between its outgoing weights and the fixed feedback weights. This invariant acts as an implicit regularizer, constraining the network's weight trajectories to a lower-dimensional manifold where alignment is actively preserved during training.
In both cases, alignment is the byproduct of a constraint: regularization carves out a restricted space where the network must live, and the only way to minimize error within that space is to align with the feedback signal.
The Geometry of Stationarity: Contractive vs. Expansive Forces
This relationship between regularization and alignment extends far beyond feedback networks. It points to a universal geometric principle governing any learning system near optimization stationarity—the point where a network has finished learning on average, and its net weight updates hover around zero.
As framed by Koplow et al., the total update applied to a network's weights can be decomposed into a tug-of-war between two opposing forces:
- Contractive Forces: Regularization mechanisms, like $L_2$ weight decay, act globally to pull weights toward the origin, shrinking their norms.
- Expansive Forces: The error-driven learning signal pushes weights outward to satisfy the performance objective and prevent the network from collapsing to zero.
◄───────────────────────────█───────────────────────────►
Contractive Force Expansive Force
(L2 Weight Decay) (Learning Signal)
Pulls weights to zero Pushes weights outward
At stationarity, these forces must balance perfectly. If a network incorporates $L_2$ weight decay, the learning signal cannot be arbitrary; it is mathematically forced to provide an equal and opposite expansive force to maintain stable weight norms Koplow et al.. This requirement introduces a severe geometric bias into the direction of the weight updates.
Why Stability Breeds Hebbian Dynamics
Surprisingly, this geometric bias looks exactly like Hebbian learning—the biological principle that "cells that fire together, wire together."
To counteract the contractive pull of weight decay and keep the weight norms stable, the learning update must push outward in a direction that correlates with the current weight state and network activations. As demonstrated by Koplow et al., this mathematical stability requirement forces the average learning update to align with the product of pre- and post-synaptic activity near stationarity.
Crucially, the authors prove that this is an emergent epiphenomenon of regularized optimization rather than a hardcoded rule Koplow et al.. Because this alignment occurs across virtually any complex update rule—even non-local or random ones—observing Hebbian patterns in a system does not prove that the system is driven by a local, correlation-based learning rule. Instead, global error-driven optimization algorithms (like backpropagation) masquerade as Hebbian learners simply because they are fighting against weight decay to maintain equilibrium.
The Noise Phase Transition
If Hebbian dynamics emerge because the learning signal must act as an expansive force, what happens when the system experiences strong stochastic noise?
Stochastic noise introduces its own random, outward fluctuations, acting as an inherent expansive force. This disrupts the equilibrium. As analyzed by Koplow et al., the system undergoes a predictable phase transition determined by the ratio of regularization strength to stochastic noise:
- High Regularization / Low Noise: Weight decay dominates the force balance. The learning signal must be expansive to keep weights from shrinking to zero, resulting in Hebbian alignment.
- Low Regularization / High Noise: Stochastic noise dominates, threatening to push weights to infinity. To maintain stability, the learning signal must flip and become contractive to rein in the noise. This geometric inversion forces the updates into an anti-Hebbian alignment.
By viewing learning rules through the lens of contractive and expansive force balancing, this framework unifies seemingly disparate phenomena. Whether explaining how forward weights track random gradients or why global optimizers look locally Hebbian, the underlying engine is the same: regularization forces the network to align with its environment to survive.
Key Foundational Work
Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks (2024) — contributed 3 concept elements to this direction.
Convergence and Alignment of Gradient Descent with Random Backpropagation Weights (2021) — contributed 2 concept elements to this direction.