Overcoming Feature Shortsightedness in Local Learning

21 concepts16 papers1 proposal

Standard deep learning relies on end-to-end backpropagation, a process that creates a strict "backward-locking" computational bottleneck. To update the weights of an early layer, a network must execute a full forward pass, compute a global loss, and propagate gradients sequentially backward through every single layer. Storing all intermediate activations for this backward journey creates a massive memory overhead, severely limiting our ability to scale up models.

Local learning offers an elegant escape. By breaking a deep network into independent blocks or layers, each module can be trained using its own local objective function (such as a local contrastive loss or an auxiliary classifier) immediately after its forward pass. This eliminates the need to store activations across the entire architecture, drastically slashing GPU memory usage.

However, decoupling these layers introduces a fundamental flaw: local learning myopia, or feature shortsightedness. When individual layers optimize in isolation, they fail to coordinate, leading to severe performance degradation compared to end-to-end training.

The Information Bottleneck of Isolated Layers

To understand why naive local learning fails, we must look at how information flows through a deep network. Deep learning succeeds because of hierarchical compositionality: early layers extract low-level primitives (like edges and textures) that deeper layers progressively assemble into complex, high-level concepts.

When we train a network end-to-end, the global gradient ensures that early layers preserve any information necessary for the final task. But when layers are optimized using purely local objectives, they act greedily. As originally observed by Yoshua Bengio et al. in early work on Greedy Layer-Wise Training of Deep Networks, a local objective can cause a layer to prematurely discard features that seem irrelevant to its immediate local task but are critical for downstream layers.

Chengting Yu et al. formalize this in Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply by viewing greedy feedforward training as a Markov chain. In this chain, task-relevant information progressively decays at each layer boundary. Once an early layer discards a piece of information to maximize its local loss, that information is permanently lost; subsequent local modules cannot recover it.

This shortsightedness manifests in several ways:

To bridge this performance gap, recent research focuses on building mechanisms that allow isolated layers to communicate and coordinate without reverting to full backpropagation.

Mechanism 1: Auxiliary Networks and Context Supply

If the primary issue is that early layers cannot anticipate what downstream layers need, one solution is to actively feed downstream context or global momentum back into the local training blocks.

[ Layer i ] ---> [ Momentum / Auxiliary Network ] ---> ( Injects Global Context )
    |                                                          |
    v                                                          v
[ Local Loss i ] <---------------------------------------------+

In Momentum Auxiliary Network for Supervised Local Learning, Junhao Su et al. introduced the Momentum Auxiliary Network (MAN). Instead of training a block in total isolation, MAN uses an Exponential Moving Average (EMA) and learnable biases to pass an architectural memory across blocks. This effectively bridges the information gap, allowing early layers to retain features that will be useful later.

Building on this concept to scale local learning to larger architectures, Yuming Zhang et al. developed the Multilaminar Leap Augmented Auxiliary Network (MLAAN) in MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Network. MLAAN utilizes a dual-layer auxiliary structure and a "leap" mechanism to share context across isolated modules, ensuring that early layers remain aware of high-level task requirements while still slashing GPU memory usage by up to 70%.

Mechanism 2: Top-Down Feedback and Backward Training

Another way to prevent information loss is to introduce explicit top-down feedback loops that run parallel to, or immediately after, the local forward updates.

For graph-structured data, Hongyuan Zhang et al. proposed Stacked Graph Neural Networks (SGNN) in Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One. To prevent the performance degradation typical of decoupled training, the authors implemented a "Backward Training" mechanism. This mechanism passes feedback from deeper layers back to shallow layers, communicating downstream requirements so that greedy local training can achieve the optimization benefits of end-to-end backpropagation.

Similarly, Namyong Park et al. designed Forward Learning of Graph Neural Networks (FORWARDGNN), which pairs local contrastive learning with top-down feedback signals. By allowing deeper layers to influence the local objectives of earlier layers, the architecture avoids the siloing effect and ensures that individual layers contribute effectively to the global task.

Mechanism 3: Gradient Reconciliation and Global Correction

A third perspective addresses the problem mathematically: if layers update themselves using entirely separate objectives, their optimization trajectories can conflict, leading to discordant updates that actively hinder adjacent layers.

End-to-End BP:      [ Layer 1 Update ] <=== aligned via chain rule ====> [ Layer 2 Update ]

Isolated Local:     [ Layer 1 Update ] (Discordant / Conflicting)       [ Layer 2 Update ]

Reconciliation:     [ Layer 1 Update ] <--- Forced Alignment (SGR) ----> [ Layer 2 Update ]

To solve this, Yibo Yang et al. developed Successive Gradient Reconciliation (SGR) in Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation. SGR forces neighboring layers to mathematically "reconcile" their local gradients during training. This alignment guarantees that the error in the final layer converges toward its true global optimum, closing the theoretical convergence gap that has historically plagued modular training.

Alternatively, instead of continuous reconciliation, some frameworks rely on periodic global alignment. In BP-Modified Local Loss for Efficient Training of Deep Neural Networks, Lianhai Ren et al. proposed a hybrid approach. The network blocks are trained independently using local losses to save memory, but the framework periodically injects a "global correction" signal. This sparse global intervention realigns the isolated blocks, combining the memory efficiency of local learning with the strict accuracy of global backpropagation.

By shifting the paradigm from isolated local learning to coordinated local learning, these diverse approaches—whether using auxiliary context, top-down feedback, or gradient alignment—successfully overcome feature shortsightedness, unlocking high-performance, memory-efficient deep learning.

Key Foundational Work

LoCo: Local Contrastive Representation Learning (2020) — contributed 1 concept elements to this direction.
Greedy Layer-Wise Training of Deep Networks (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_inter_layer_coordination_gap_ecfc27d7,
  title = {inter-layer coordination gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inter-layer-coordination-gap/rdr_ecfc27d7a1874666a66ca609c84f4f35},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_inter_layer_coordination_gap_ecfc27d7_v1,
  title = {inter-layer coordination gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inter-layer-coordination-gap/rdr_ecfc27d7a1874666a66ca609c84f4f35/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake