Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply

Chengting Yu, Fengzhao Zhang, Hanzhi Ma, Aili Wang, Erping LiSubmission to ICLR 2024 · Withdrawn

TL;DR

Greedy Local Learning (GLL) divides deep networks into gradient-isolated modules to slash GPU memory and unlock parallel training, but its accuracy plunges when splitting into many modules. This paper identifies an information-theoretic bottleneck behind this collapse—the "confirmed habits" dilemma—and introduces ContSup (Context Supply), a lightweight mechanism that injects input or skip-layer context directly into downstream modules to restore lost task-relevant information.

What Problem Does This Paper Address?

Standard deep networks rely on end-to-end (E2E) backpropagation, which suffers from the backward-locking problem: activations for every layer must remain in GPU memory until the backward pass reaches them. This severely caps batch sizes and input resolutions, while preventing inter-layer parallelization.

Greedy Local Learning (GLL) addresses this by breaking a network into $K$ gradient-isolated blocks, each trained using its own local auxiliary loss. While GLL slashes peak GPU memory, it suffers from a steep drop in accuracy as the number of modules $K$ increases. Prior fixes attempted to force intermediate representations to reconstruct the input (preserving information entropy), but they require heavy decoder heads and still suffer severe degradation when $K$ is large.

Methods

The authors analyze GLL through the lens of Information Theory and the Data Processing Inequality:

  • The "Confirmed Habits" Dilemma: In standard GLL, feature propagation forms a Markov chain: $(y, r) \to x \to \dots \to h_{l-1} \to h_l \to \hat{y}_l$. Because early local classifiers greedily optimize only for their immediate objective, task-relevant mutual information $I(h_l, y)$ monotonically decreases with depth. Once discarded by an early module, this information is lost forever to subsequent modules.
  • Context Supply (ContSup): To break the Markov chain, ContSup injects external context $c_l$ into intermediate features prior to module processing: $$h_{l-1}^c = h_{l-1} + M_l(c_l)$$ where $M_l$ is a lightweight module-specific alignment function.

The authors evaluate several context configurations:

  • ContSup[E]: Feeds the raw input $x$ through a compact 2-convolution local encoder $M_E(x)$.
  • ContSup[R1]: Adds a shortcut connection from the preceding module's input feature ($h_{l-2}^c$) using a $1\times 1$ convolution $M_R$.
  • ContSup[R1E]: Combines both the input encoder and adjacent shortcut connection.
  • ContSup[RnE]: Extends the concept to multi-hop topological skip connections across prior modules.
       Input (x) ──────────────────────────┐
          │                                 ▼
          ▼                           [Local Encoder]
     [ Module I ] ──► h1 ──(+)──────► [ Module II ] ──► h2 ...
                              ▲
     (Optional Shortcut) ─────┘

Datasets

The authors evaluated their method on standard vision benchmarks:

  • CIFAR-10 (60,000 $32\times32$ color images, 10 classes)
  • SVHN (Street View House Numbers; 73,257 train / 26,032 test images, 10 digit classes)
  • STL-10 ($96\times96$ color images, 10 classes)

Note: No new datasets were introduced.

Key Insights and Ideas

  • Why reconstruction losses hit a ceiling: Existing methods like InfoPro train decoders to maximize $I(h_l, x)$, preserving input information. However, the theoretical upper bound is still restricted by the prior module's feature: $I(h_l, y) \le I(h_{l-1}, y)$. Decoding cannot recover task-relevant signals that were already eliminated upstream.
  • Bypassing the Data Processing Inequality: Providing direct context bypasses the strict cascade assumption. Because context feeds fresh task information into deeper layers, $I(h_l, y) \ge I(h_{l-1}, y)$ becomes possible.
  • Lean encoders beat heavy decoders: Reconstructing high-dimensional inputs requires large auxiliary decoders and delicate loss balancing. In contrast, projecting the input forward into module latent space via a small encoder adds negligible GPU memory while directly replenishing representation capacity.

Key Results

  • SOTA Accuracy under Fine Partitioning: On CIFAR-10 using ResNet-32 split into $K=16$ isolated modules:

  • Baseline Decoupled Greedy Learning (DGL): 16.21% error

  • InfoPro (Contrastive): 12.75% error
  • BackLink: 12.4% error
  • ContSup[R1E] (Contrastive): 9.98% error (closing the gap toward E2E's 6.50% on ResNet-110).
  • Consistent Gains Across Benchmarks: On ResNet-110 with $K=16$, ContSup[R1E] cuts CIFAR-10 test error from 13.80% (DGL) down to 8.89%, SVHN error from 5.55% down to 3.91%, and STL-10 error from 27.73% down to 23.38%.
  • Memory and Training Efficiency:
  • ContSup drastically reduces GPU memory usage compared to full E2E training while achieving competitive accuracy.
  • Training speed is notably faster than InfoPro (e.g., $1.23\text{ s}$ vs. $1.60\text{ s}$ per batch on ResNet-32, $K=16$) because it avoids heavy decoder backpropagation.

Limitations and Future Work

  • Inference Overhead: Unlike standard GLL where auxiliary training heads are discarded, ContSup’s context-encoding paths must remain during inference, increasing test-time latency by roughly 8% to 16% on heavily partitioned models ($K=16$).
  • Heuristic Context Selection: Context pathways are currently based on fixed structural priors (raw input and adjacent shortcuts). Principled pruning and automated topological routing remain open challenges.
  • Diminishing Gains at Low $K$: When the network is split into very few modules ($K=2$ or $4$), the confirmed habit dilemma is minimal, yielding modest gains over existing baselines.

Summary updated

Abstract

Traditional end-to-end (E2E) training of deep networks necessitates storing intermediate activations for back-propagation, resulting in a large memory footprint on GPUs and restricted model parallelization. As an alternative, greedy local learning partitions the network into gradient-isolated modules and trains supervisely based on local preliminary losses, thereby providing asynchronous and parallel training methods that substantially reduce memory cost. However, empirical experiments reveal that as the number of segmentations of the gradient-isolated module increases, the performance of the local learning scheme degrades substantially, severely limiting its expansibility. To avoid this issue, we theoretically analyze the greedy local learning from the standpoint of information theory and propose a ContSup scheme, which incorporates context supply between isolated modules to compensate for information loss. Experiments on benchmark datasets (i.e. CIFAR, SVHN, STL-10) achieve SOTA results and indicate that our proposed method can significantly improve the performance of greedy local learning with minimal memory and computational overhead, allowing for the boost of the number of isolated modules.
BibTeX
@article{yu2025go,
  author = {Chengting Yu and Fengzhao Zhang and Hanzhi Ma and Aili Wang and Erping Li},
  title = {{Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2025},
  doi = {10.1109/tai.2025.3528384},
}
Figure