TL;DR
Greedy Local Learning (GLL) divides deep networks into gradient-isolated modules to slash GPU memory and unlock parallel training, but its accuracy plunges when splitting into many modules. This paper identifies an information-theoretic bottleneck behind this collapse—the "confirmed habits" dilemma—and introduces ContSup (Context Supply), a lightweight mechanism that injects input or skip-layer context directly into downstream modules to restore lost task-relevant information.
What Problem Does This Paper Address?
Standard deep networks rely on end-to-end (E2E) backpropagation, which suffers from the backward-locking problem: activations for every layer must remain in GPU memory until the backward pass reaches them. This severely caps batch sizes and input resolutions, while preventing inter-layer parallelization.
Greedy Local Learning (GLL) addresses this by breaking a network into $K$ gradient-isolated blocks, each trained using its own local auxiliary loss. While GLL slashes peak GPU memory, it suffers from a steep drop in accuracy as the number of modules $K$ increases. Prior fixes attempted to force intermediate representations to reconstruct the input (preserving information entropy), but they require heavy decoder heads and still suffer severe degradation when $K$ is large.
Methods
The authors analyze GLL through the lens of Information Theory and the Data Processing Inequality:
- The "Confirmed Habits" Dilemma: In standard GLL, feature propagation forms a Markov chain: $(y, r) \to x \to \dots \to h_{l-1} \to h_l \to \hat{y}_l$. Because early local classifiers greedily optimize only for their immediate objective, task-relevant mutual information $I(h_l, y)$ monotonically decreases with depth. Once discarded by an early module, this information is lost forever to subsequent modules.
- Context Supply (ContSup): To break the Markov chain, ContSup injects external context $c_l$ into intermediate features prior to module processing: $$h_{l-1}^c = h_{l-1} + M_l(c_l)$$ where $M_l$ is a lightweight module-specific alignment function.
The authors evaluate several context configurations:
ContSup[E]: Feeds the raw input $x$ through a compact 2-convolution local encoder $M_E(x)$.ContSup[R1]: Adds a shortcut connection from the preceding module's input feature ($h_{l-2}^c$) using a $1\times 1$ convolution $M_R$.ContSup[R1E]: Combines both the input encoder and adjacent shortcut connection.ContSup[RnE]: Extends the concept to multi-hop topological skip connections across prior modules.
Input (x) ──────────────────────────┐
│ ▼
▼ [Local Encoder]
[ Module I ] ──► h1 ──(+)──────► [ Module II ] ──► h2 ...
▲
(Optional Shortcut) ─────┘
Datasets
The authors evaluated their method on standard vision benchmarks:
- CIFAR-10 (60,000 $32\times32$ color images, 10 classes)
- SVHN (Street View House Numbers; 73,257 train / 26,032 test images, 10 digit classes)
- STL-10 ($96\times96$ color images, 10 classes)
Note: No new datasets were introduced.
Key Insights and Ideas
- Why reconstruction losses hit a ceiling: Existing methods like InfoPro train decoders to maximize $I(h_l, x)$, preserving input information. However, the theoretical upper bound is still restricted by the prior module's feature: $I(h_l, y) \le I(h_{l-1}, y)$. Decoding cannot recover task-relevant signals that were already eliminated upstream.
- Bypassing the Data Processing Inequality: Providing direct context bypasses the strict cascade assumption. Because context feeds fresh task information into deeper layers, $I(h_l, y) \ge I(h_{l-1}, y)$ becomes possible.
- Lean encoders beat heavy decoders: Reconstructing high-dimensional inputs requires large auxiliary decoders and delicate loss balancing. In contrast, projecting the input forward into module latent space via a small encoder adds negligible GPU memory while directly replenishing representation capacity.
Key Results
-
SOTA Accuracy under Fine Partitioning: On CIFAR-10 using ResNet-32 split into $K=16$ isolated modules:
-
Baseline Decoupled Greedy Learning (DGL): 16.21% error
- InfoPro (Contrastive): 12.75% error
- BackLink: 12.4% error
- ContSup[R1E] (Contrastive): 9.98% error (closing the gap toward E2E's 6.50% on ResNet-110).
- Consistent Gains Across Benchmarks: On ResNet-110 with $K=16$, ContSup[R1E] cuts CIFAR-10 test error from 13.80% (DGL) down to 8.89%, SVHN error from 5.55% down to 3.91%, and STL-10 error from 27.73% down to 23.38%.
- Memory and Training Efficiency:
- ContSup drastically reduces GPU memory usage compared to full E2E training while achieving competitive accuracy.
- Training speed is notably faster than InfoPro (e.g., $1.23\text{ s}$ vs. $1.60\text{ s}$ per batch on ResNet-32, $K=16$) because it avoids heavy decoder backpropagation.
Limitations and Future Work
- Inference Overhead: Unlike standard GLL where auxiliary training heads are discarded, ContSup’s context-encoding paths must remain during inference, increasing test-time latency by roughly 8% to 16% on heavily partitioned models ($K=16$).
- Heuristic Context Selection: Context pathways are currently based on fixed structural priors (raw input and adjacent shortcuts). Principled pruning and automated topological routing remain open challenges.
- Diminishing Gains at Low $K$: When the network is split into very few modules ($K=2$ or $4$), the confirmed habit dilemma is minimal, yielding modest gains over existing baselines.
Summary updated