Continual Learning Under Strict Resource Constraints

108 concepts99 papers1 proposal

Continual learning (CL) promises neural networks that can adapt to non-stationary data streams without catastrophically forgetting the past. However, as noted by Liyuan Wang et al. in their comprehensive survey, this long-term adaptation introduces significant computational and storage overhead. Traditional CL strategies rely on expanding network architectures, maintaining massive experience replay buffers, or training complex generative models to replay past data.

While these methods perform well in unconstrained environments, they hit a wall when deployed on edge hardware, microcontrollers, or high-throughput streams. The core challenge shifts from how to remember to how to remember efficiently.

Shifting the Bottleneck from Storage to Compute

Historically, CL research treated physical storage as the primary bottleneck. However, Ameya Prabhu et al. demonstrated that in real-world high-throughput environments, the true constraint is computation time. Under a strict computational budget, complex state-of-the-art CL algorithms often underperform compared to a simple uniform sampling baseline. This occurs because sophisticated algorithms pay a "simplicity tax"—they spend valuable clock cycles computing gradient projections or optimizing data selection, leaving insufficient resources for basic model convergence. Uniform sampling, being computationally free, allows the model to process significantly more data within the same time window.

Similarly, James Seale Smith et al. argue that for modern foundation models, the bottleneck is compute rather than storage capacity. To formalize this paradigm, Saurabh Kumar et al. frame continual learning as a computationally constrained reinforcement learning problem. In this view, constraints on per-timestep computation fundamentally dictate that an agent must selectively learn and forget, transforming resource limitations from a practical afterthought into a core algorithmic driver.

Bypassing Backpropagation via Embedding Spaces

One of the most effective ways to eliminate computational overhead is to avoid updating the entire deep network altogether. In Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis, Christopher Kanan et al. introduce Deep SLDA, which pairs a frozen deep feature extractor with an incrementally updated Linear Discriminant Analysis (LDA) classifier. Because the feature extractor is fixed, the model requires no backpropagation for feature learning. Instead, it shifts the problem to online statistical updates—tracking running class means and covariances—allowing it to run orders of magnitude faster and use drastically less memory than standard rehearsal methods.

Operating entirely within the embedding space offers a powerful alternative to raw data storage. Jary Pomponi et al. propose Centroids Matching, an approach that represents learned classes as geometric centroids in a fixed embedding space. Instead of storing individual historical samples or enforcing expensive optimization constraints, the model prevents forgetting simply by preserving the geometric relationships of previously learned features as new data arrives.

Sparsity and Gating for Extreme Edge Hardware

When models must learn directly on extreme edge hardware like microcontrollers (MCUs), even partial backpropagation can overwhelm the hardware's static memory. To address this, Francesco Paissan et al. combine latent replay with structured sparse backpropagation. By updating only a carefully selected subset of weights during backward passes, they drastically reduce the memory footprint of on-device training, enabling low-power devices to acquire new categories in minutes.

Rather than explicitly computing and storing complex parameter masks for every task—which Mengqi Xue et al. note creates prohibitive memory bloat—efficient architectures leverage implicit or lightweight gating. For instance, Iman Mirzadeh et al. demonstrate that standard Dropout acts as an implicit gating mechanism that naturally segregates task representations, preventing catastrophic forgetting without adding any computational or storage overhead.

For architectures requiring explicit routing, single-author works provide minimalist solutions: Prashant Shivaram Bhat introduces tiny, task-specific attention bottlenecks to route information through a shared representation space. This minimizes the resource footprint while scaling gracefully to long sequences of tasks, proving that resource-constrained continual learning relies not on storing more, but on processing smarter.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continual_learning_overhead_e4b5a223,
  title = {continual learning overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continual-learning-overhead/rdr_e4b5a2234a49491297f778ea6a275143},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_continual_learning_overhead_e4b5a223_v1,
  title = {continual learning overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continual-learning-overhead/rdr_e4b5a2234a49491297f778ea6a275143/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake