The Core Problem: Representation Drift
In traditional continual learning, a neural network updates its weights sequentially as it encounters new tasks or classes. While this allows the network to adapt to new data, backpropagation inevitably warps the underlying feature space. This phenomenon is known as representation drift or feature drift.
When the feature space shifts to accommodate new categories, the geometric boundaries previously established for older tasks collapse. This is the root cause of catastrophic forgetting. Historically, researchers attempted to mitigate this drift by regularizing weight updates or maintaining complex episodic replay buffers. However, these methods often introduce severe computational overhead or merely delay the onset of forgetting.
The Frozen Backbone Paradigm
An elegant alternative has emerged: if a feature space is already robust and general enough, we can eliminate representation drift entirely by freezing the feature extraction backbone.
Instead of treating continual learning as an ongoing optimization problem across the entire network, this paradigm splits the architecture into a rigid, invariant feature extractor and a highly plastic downstream adaptation layer. By keeping the backbone weights completely static, the model guarantees absolute stability for previously learned representations.
This approach has become highly viable due to the rise of large-scale foundation models and self-supervised pre-training. As Liu et al. demonstrate, simple baselines built on frozen, self-supervised pre-trained models can outperform complex, state-of-the-art continual learning algorithms. Because modern backbones are trained on massive, diverse datasets, they provide a highly generalizable feature space that naturally captures the essential textures, boundaries, and semantics required for novel tasks without needing further fine-tuning (Pian et al.; Zhuang et al.).
Mechanisms of Downstream Plasticity
If the underlying features cannot change, how does the model learn new concepts? Researchers have developed several distinct strategies to achieve plasticity purely within the downstream classification and projection layers.
Geometric and Prototype-Based Classifiers
Instead of training parametric decision boundaries that can drift over time, many frozen-backbone architectures rely on non-parametric or prototype-based classification. For example, Ayub et al. introduce Centroid-Based Concept Learning (CBCL), which computes and stores statistical centroids for new classes within the invariant feature space. Because the backbone is frozen, these centroids never shift. To handle evolving boundaries between old and new classes, Zhang et al. decouple the network by pairing a frozen backbone with a Graph Attention Network that dynamically updates only the classification layer, allowing boundaries to adapt smoothly without risking representation corruption.
High-Dimensional Projections
To make frozen features more linearly separable for novel tasks, some frameworks project the static embeddings into a much larger, sparser space. Hersche et al. leverage hyperdimensional computing to map frozen features into a high-dimensional space of quasi-orthogonal vectors. This allows the model to scale to thousands of incremental classes with zero backbone retraining. Similarly, McDonnell et al. introduce RanPAC, which passes frozen backbone features through a fixed random projection layer before feeding them into an analytical classifier, achieving state-of-the-art continual learning performance with minimal computational overhead.
Parameter-Efficient Fine-Tuning (PEFT)
When downstream tasks require structural or contextual adaptation beyond simple classification, researchers inject tiny, isolated pockets of plasticity directly into or alongside the frozen backbone:
- Visual Prompting: Kim et al. propose ECLIPSE, which keeps the base model frozen and tunes task-specific visual prompts (accounting for just 1.3% of total parameters) to handle incremental panoptic segmentation.
- Subspace Fine-Tuning: Rather than updating entire weight matrices, Wang et al. introduce Singular Value Fine-tuning (SVFCL). By freezing the singular vectors of a foundation model's weights and updating only the singular values, they allow the model to adapt to new classes while ensuring the underlying representational basis remains perfectly anchored.
- Task-Specific Adapters: In sequential robotics tasks, Liu et al. utilize Low-Rank Adaptation (LoRA) layers alongside a frozen encoder, preserving the foundational visual features while allowing the policy to acquire a continuous stream of new manipulation skills.
The Memory Advantage: Eliminating Activation Staleness
Beyond preventing classifier degradation, a frozen backbone solves a critical bottleneck in memory-augmented and exemplar-based continual learning systems: activation staleness.
In traditional continual learning systems that utilize an external episodic memory, storing raw data (like images or text) is incredibly expensive, so systems often cache intermediate feature vectors instead. However, if the backbone weights are updated during a new task, all previously cached feature vectors instantly become obsolete because they were generated by an older version of the network.
By enforcing a static backbone, cached representations remain permanently valid and perfectly compatible with the network's current state. For instance, Wang et al. exploit this in their LONGMEM architecture, using a frozen pre-trained model as a dedicated memory encoder. Because the encoder is invariant, long-term stored vector representations never suffer from representation drift, allowing a separate, lightweight network to reliably retrieve and utilize historical context across indefinite horizons.