When a neural network learns a new task, it often overwrites the weights used for previous tasks, a phenomenon known as "catastrophic forgetting." The most common remedy is Experience Replay: storing a buffer of old examples and periodically re-showing them to the model. However, storing raw data—like high-resolution images or complex graph structures—is incredibly expensive in terms of both memory and the computation required to re-process those samples from scratch.
A growing body of research suggests a more efficient alternative: instead of replaying raw inputs, we should replay the latent feature embeddings generated by the model's intermediate layers.
The Memory Bottleneck of Raw Replay
In environments like autonomous driving or high-resolution image segmentation, raw data is massive. Marin Toromanoff, Fabien Moutarde, and Émilie Wirbel demonstrated that storing low-dimensional feature vectors instead of raw images allows reinforcement learning agents to handle much higher input resolutions. This is critical for detecting small, distant objects like traffic lights that would be lost if the raw images were downsampled to fit in a standard memory buffer.
By compressing high-dimensional visual data into "abstract observations," researchers like Hyung-Jin Yoon have shown that agents can store millions of training steps on standard hardware, making off-policy learning feasible in visually complex environments. In the context of large-scale models like the Segment Anything Model (SAM), Kangye Ji and colleagues found that replaying latent embeddings can reduce buffer requirements by up to 256x compared to traditional image-based buffers.
Skipping Redundant Computation
Latent replay doesn't just save space; it saves time. When we replay a raw image, the model must perform a full forward and backward pass through every single layer. However, the early layers of a deep network often act as general-purpose feature extractors that change very little once they have seen enough data.
Оleksiy Ostapenko, Timothée Lesort, and their team found that by using a "frozen" foundation model as a feature extractor and replaying the resulting latents, they could achieve a 100x increase in compute efficiency. Because the early layers are frozen, the system bypasses the redundant processing of those layers during rehearsal. This approach, also championed by Soumya Banerjee and Vinay P. Namboodiri in their work on "Streaming LifeLong Learning," allows models to learn from a continuous, one-pass stream of data while maintaining the ability to perform "any-time inference."
Why Latents are Better Than Pixels
Beyond efficiency, replaying latents may actually be more effective for maintaining model stability. Raw data is noisy and contains many details irrelevant to the task at hand. In contrast, the model's internal embedding space captures the structural and semantic relationships it has already learned.
Fan Zhou and Chengtai Cao discovered this when applying experience replay to Graph Neural Networks (GNNs). They found that selecting "representative" data points for the memory buffer is significantly more effective when done within the learned embedding space rather than the raw attribute space. By choosing samples that are central or unique in the latent space, the model ensures that the most informative "memories" are preserved, helping the GNN maintain performance across a sequence of evolving graph tasks.
Summary of the Intuition
The shift from raw replay to latent replay rests on three pillars:
- Compression: Latent vectors are orders of magnitude smaller than raw inputs, allowing for much larger and more diverse memory buffers.
- Truncation: By replaying at an intermediate layer, we skip the "front-end" of the network, drastically reducing the FLOPs required for each training step.
- Abstraction: Latent spaces filter out noise, allowing the model to focus on the semantic features that actually prevent forgetting.
This framework transforms continual learning from a heavy, resource-intensive process into a lightweight mechanism suitable for real-time streaming and constrained robotic systems.