Sublinear Memory Scaling for Continual Learning

105 concepts97 papers1 proposal

When a neural network learns tasks sequentially, it suffers from catastrophic forgetting: training on new data erases the weight configurations required for older tasks. As Kemker et al. demonstrate, rehearsal-based methods—which store a subset of historical data and periodically replay it during training—are exceptionally effective at preserving past knowledge compared to architectural or regularization-based alternatives.

However, traditional rehearsal introduces a severe bottleneck: linear memory scaling ($O(T)$). As the number of tasks ($T$) or classes grows indefinitely, storing raw data, model snapshots (Hess et al.), or task-specific parameter prompts (Tiwari et al.) creates an unsustainable storage overhead. This linear growth restricts lifelong learning systems from operating on resource-constrained hardware or over infinite horizons.

To build scalable continual learning systems, researchers are moving away from raw data storage toward sublinear or constant-size memory mechanisms.


The Dilution Dilemma of Fixed Buffers

A naive solution to linear memory growth is to enforce a fixed-size replay buffer. However, this introduces a trade-off: as new tasks arrive, the buffer must discard older samples to maintain its size.

As Mi et al. and Febrinanto et al. point out, a fixed buffer causes the number of representative exemplars allocated to each individual task to diminish toward zero over time. This dilution severely degrades the model's ability to preserve early knowledge. Conversely, allowing the buffer to expand unconstrained causes training energy consumption to scale quadratically (Trinci et al.) and creates computational bottlenecks during retrieval (Yang et al.).

To break this bottleneck, modern frameworks compress historical knowledge into constant-size spaces using three primary strategies: generative pseudo-replay, subspace compression, and bounded state summaries.


Strategy 1: Generative Pseudo-Replay

Instead of storing raw pixels, tokens, or graph structures (Downer et al.), generative pseudo-replay trains a secondary generative model (such as a VAE or GAN) alongside the primary task network. When a new task arrives, the generator synthesizes "pseudo-samples" of past tasks, which are mixed with the new data stream.

[New Task Data] ───┐
                   ▼
[Generator] ──► [Pseudo-Samples] ──► [Joint Training] ──► [Updated Model]

Why it works

A generative model's parameter footprint is fixed. By learning the distribution of past data rather than archiving individual instances, the system compresses an infinite history of experiences into a constant number of weights.

Rostami et al. and Rostami demonstrate that generative autoencoders can successfully align sequential tasks within a shared, task-invariant latent embedding space. This allows the network to consolidate memory without explicit data storage. Similarly, Binici et al. apply this concept to data-free knowledge distillation, using a VAE to mimic synthetic historical data within a tiny, constant memory footprint. To prevent the generator itself from forgetting, Ye et al. introduce a twin-generator framework where two identical models alternately act as teacher and assistant, passing historical knowledge forward without storing raw data.


Strategy 2: Compressing Parameter and Function Spaces

An alternative to replaying data is to protect the mathematical directions the model has already optimized. Gradient projection methods identify the subspace of weights critical to past tasks and constrain future updates to be strictly orthogonal to them. However, storing raw activation bases or curvature matrices for every task also scales linearly.

Why it works

Instead of storing uncompressed task bases, systems can compute compact statistical summaries or low-rank approximations of the parameter space.

For example, Ritter et al. use a Kronecker-factored Laplace approximation to capture weight interactions within layers. This allows the model to retain a structured memory of curvature without storing separate matrices for every task.

In large language models, where gradient buffers are massive, Nayak et al. employ adaptive Singular Value Decomposition (SVD) to identify and freeze critical weight directions. By constraining new updates to the orthogonal complement of these directions, they eliminate the linear growth of activation bases seen in older projection methods like those discussed by Lin et al..

In function space, Titsias et al. use sparse Gaussian Processes. Instead of archiving distinct summaries for every learned task, they optimize a fixed number of "inducing points" that summarize what the model outputs should look like, consolidating functional memory into a constant-size buffer.


Strategy 3: Bounded State Summaries and Hierarchies

For online meta-learning and reinforcement learning, agents must track non-stationary environments over time. Storing entire trajectories or historical observations creates an unsustainable memory burden as the operational horizon approaches infinity (Lee et al.).

Why it works

By replacing perfect data recall with dynamic task boundary detection or hierarchical indexing, models can achieve sublinear or constant $O(1)$ memory scaling.

The online meta-learning framework proposed by Sow et al. achieves a constant $O(1)$ memory footprint by using statistical mechanisms to detect task switches and distribution shifts in real time. Instead of archiving past task data to prevent forgetting, the algorithm updates a fixed-size state vector that encapsulates historical trajectory information.

Linear Scaling (Standard):      [Task 1 Data] + [Task 2 Data] + [Task 3 Data] ──► O(T) Memory
Constant Scaling (Sublinear):   [Data Stream] ──► [Boundary Detector] ──► [O(1) State Vector]

When structural expansion is necessary, it can be optimized to scale sublinearly. Masarczyk et al. introduce an architecture that organizes generative models into a binary-indexed hierarchy. By mimicking the logic of binary addition, this approach ensures that both the memory footprint and the required retraining steps scale logarithmically ($O(\log T)$) rather than linearly with the number of tasks.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_memory_scaling_experience_replay__a70624bf,
  title = {linear memory scaling (experience replay scalability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-memory-scaling-experience-replay-scalability/rdr_a70624bf1f534091942b410a9a99ddda},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_memory_scaling_experience_replay__a70624bf_v1,
  title = {linear memory scaling (experience replay scalability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-memory-scaling-experience-replay-scalability/rdr_a70624bf1f534091942b410a9a99ddda/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake