The Scalability of Biologically Plausible Learning

89 concepts68 papers1 proposal

Deep learning owes its extraordinary success to backpropagation (BP). By calculating exact error gradients across a global computational graph, BP allows deep networks to coordinate millions of parameters to solve complex, high-dimensional tasks. Yet, from a neuroscientific perspective, BP is fundamentally flawed. It requires a physically impossible mechanism known as the weight transport problem: the feedback pathways must know the exact transpose of the feedforward weight matrices ($W^T$) to propagate gradients. Furthermore, BP relies on distinct, synchronized forward and backward phases, requiring neurons to freeze their activity while waiting for error signals to arrive.

To bridge this gap, researchers have proposed various "biologically plausible" learning rules that rely only on local information available to individual neurons or synapses. However, these alternatives have historically suffered from a severe scaling gap. Algorithms that easily solve toy problems like MNIST frequently collapse or fail to beat random baselines when applied to high-dimensional, real-world benchmarks like ImageNet or complex architectures like Transformers (Bartunov et al.; Ren et al.).

Understanding why this scaling gap exists—and how modern architectures are beginning to overcome it—is one of the most active frontiers at the intersection of AI and neuroscience.


Bypassing Weight Transport via Feedback Alignment

If a network cannot transport the exact feedforward weights to the feedback pathway, what happens if we simply use a fixed, random matrix $B$ to propagate errors instead? This is the core intuition behind Feedback Alignment (FA).

Backpropagation:       Error ---> [ Wᵀ ] ---> Local Gradient
Feedback Alignment:    Error ---> [ B  ] ---> Local Gradient (B is random & fixed)

Intuitively, it seems like random feedback weights should send meaningless garbage backward. However, during training, a remarkable phenomenon occurs: the forward weights $W$ implicitly regularize and adapt themselves to align with the fixed random matrix $B$. Once $W$ and $B$ are sufficiently aligned, the error signals flowing through $B$ begin to provide useful directional updates for the forward weights.

While early critics dismissed FA as a toy method, Launay et al. demonstrated that Direct Feedback Alignment (DFA)—a variant where the global error is broadcast directly to every hidden layer in parallel—can scale to modern deep learning architectures like Transformers and Graph Neural Networks, achieving performance remarkably close to standard backpropagation.

Despite this success, scaling FA mechanisms to high-dimensional multi-class datasets remains challenging due to their underlying optimization dynamics. Girotti et al. provided a rigorous convergence analysis showing that FA is highly sensitive to initialization. It can trigger an "implicit anti-regularization" phase where the network temporarily learns noise before aligning with the signal. To stabilize this process on complex datasets, Robertson et al. derived a mathematical conservation law showing that explicitly maximizing the alignment between forward and feedback pathways is essential to prevent representation collapse in multi-class classification.

Alternatively, the weight transport problem can be mitigated by relaxing the precision requirement. Xiao et al. demonstrated that sign-symmetry—where feedback connections do not match the exact values of forward weights but merely share their sign (positive or negative)—is sufficient to scale deep networks to massive datasets like ImageNet and MS COCO, narrowing the performance gap significantly.


Target Propagation and the Jacobian Bridge

Instead of propagating error gradients backward, an alternative strategy is to propagate target activations. This is the premise of Target Propagation (TP). If a layer $i+1$ has a desired target state that minimizes the global loss, an auxiliary feedback network can be trained to invert the forward mapping and calculate a corresponding local target for layer $i$.

Forward Pass:  Layer i ------[ W ]-----> Layer i+1
Target Prop:   Target i <----[ F ]------ Target i+1  (F tries to invert W)

The historical failure of TP to scale to deep networks stems from the difficulty of learning accurate inverse mappings. If the feedback weights $F$ do not precisely invert the forward layer, the propagated targets quickly degrade into noise as they travel deeper into the network (Ahmad et al.).

To fix this, Difference Target Propagation (DTP) introduces a correction term to account for the reconstruction error of the feedback loop. To truly bridge the scaling gap on complex visual tasks, Ernoult et al. proved that the feedback weights must learn the transpose Jacobian of the feedforward layers. By explicitly optimizing the feedback loops to approximate this geometric relationship, DTP can scale effectively to challenging benchmarks like CIFAR-10 and downsampled ImageNet.


Decomposing Networks with Local Greedy Losses

Another way to eliminate the biologically unrealistic backward pass is to avoid propagating information backward entirely. Instead of optimizing a single global loss function, a network can be broken down into independent blocks, each optimized by its own local loss function.

Global Backprop:  Input ---> [Block 1] ---> [Block 2] ---> [Global Loss]
                                 │              │               │
                                 └<───────(Backward Pass)───────┘

Local Learning:   Input ---> [Block 1] ---> [Block 2] ---> [Classifier]
                                 │              │
                           [Local Loss 1] [Local Loss 2]

The primary challenge with local learning is its "greedy" nature. Because early layers cannot see the global objective, they optimize purely for immediate, short-term metrics. This often leads to information bottlenecks where features necessary for downstream classification are discarded too early.

To overcome this optimization efficiency gap, Ren et al. introduced Local Greedy Forward Gradient, which combines activity perturbation with thousands of localized losses. This approach allows layers to update their weights during the forward pass while maintaining competitive performance on ImageNet.

In the self-supervised domain, Siddiqui demonstrated that full end-to-end backpropagation is not strictly necessary for high-dimensional feature extraction. By training individual blocks of a ResNet-50 independently using a joint-embedding self-supervised loss (Barlow Twins), the author achieved a 70.48% top-1 accuracy on ImageNet—falling just 1.1% short of standard end-to-end training. This proves that local objectives, when paired with the right self-supervised criteria, can coordinate deep hierarchies without global error propagation.


Compartmental Neurons and Continuous-Time Learning

While local losses and feedback alignment provide algorithmic workarounds, how might actual biological circuits implement these approximations? Standard artificial neurons are point-like models that compress all inputs into a single scalar activation, making it impossible to separate feedforward data from feedback errors without distinct temporal phases.

Biological pyramidal neurons solve this via their three-compartment dendritic structure. They receive feedforward inputs at the basal dendrites, feedback errors at the apical dendrites, and combine them at the soma.

                 Apical Dendrites  (Receives Top-Down Feedback / Errors)
                        │
                        ▼
                      [Soma]       (Integrates and Fires)
                        ▲
                        │
                  Basal Dendrites  (Receives Bottom-Up Inputs)

Sacramento et al. demonstrated that these distinct compartments allow circuits to calculate local prediction errors automatically. By leveraging inhibitory interneurons to cancel out expected top-down feedback, these dendritic microcircuits can mathematically approximate backpropagation in continuous time, eliminating the need for separate forward and backward phases.

To expand this to more complex architectures, Lv et al. developed Dendritic Localized Learning (DLL). DLL simultaneously satisfies three strict criteria for biological plausibility:

  1. Weight asymmetry (no weight transport)
  2. Local error representation (errors computed within dendritic compartments)
  3. Simultaneous inference and training (no phase freezing)

By mapping these constraints directly onto compartmental models, the authors successfully trained Multi-Layer Perceptrons, Convolutional Networks, and Recurrent Neural Networks to levels competitive with standard deep learning.


The Remaining Scaling Frontier

Despite these theoretical breakthroughs, a performance ceiling remains when moving from static image classification to more complex domains. Purely local Hebbian mechanisms and unsupervised similarity matching still struggle to capture the global optimization efficiency required for high-dimensional generative tasks or complex temporal dependencies (Goupy; Pehlevan).

When architectures scale down to very low unit counts—such as bio-inspired few-shot models—they often lose their efficiency advantages when faced with high-dimensional input spaces (Delahunt et al.). Similarly, biologically inspired generative frameworks, such as hippocampal Helmholtz machines that alternate between wake and sleep phases using theta oscillations, face significant scalability hurdles when processing naturalistic, high-dimensional sensory streams compared to modern machine learning models (George et al.).

To close the remaining gap, modern frameworks are increasingly focusing on hybrid approaches. For instance, Dellaferrera et al. introduced PEPITA, which uses a two-pass forward scheme where the global error modulates the input activity directly, bypassing the backward pass while preserving high-dimensional classification performance. Similarly, Erdogan et al. proposed Error Broadcast and Decorrelation (EBD), grounding learning in the mathematical principle that an optimal estimator's error must be uncorrelated with its internal states.

By moving away from strict copies of backpropagation and focusing instead on the underlying mathematical principles of alignment, decorrelation, and compartmentalized error computation, the field is steadily proving that biologically plausible learning is not a toy concept—it is a scalable alternative for the future of neuromorphic and decentralized AI hardware.

Key Foundational Work

An Information Maximization Approach to Overcomplete and Recurrent Representations (2000) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_biologically_plausible_learning_scalabil_60b9b3c7,
  title = {biologically plausible learning scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/biologically-plausible-learning-scalability/rdr_60b9b3c7da2549cbb688a52d056e528a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_biologically_plausible_learning_scalabil_60b9b3c7_v1,
  title = {biologically plausible learning scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/biologically-plausible-learning-scalability/rdr_60b9b3c7da2549cbb688a52d056e528a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake