In the standard paradigm of machine learning, models are trained on a fixed set of classes all at once. However, in continual learning, a model must master new categories sequentially. The primary obstacle to this is "catastrophic forgetting," where learning new information erodes the model's ability to recognize old concepts. While this is often discussed as a loss of weights, recent research suggests the problem is fundamentally geometric: the model’s internal feature space becomes "crowded," leading to representation overlap and interference.
The Bottleneck of Initial Success
A core reason models forget is how they utilize their embedding space during the very first phase of training. When a model learns its initial set of classes, it often utilizes the entire high-dimensional embedding space to maximize performance on those specific tasks. As Bai et al. argue, this lack of diversity in the initial feature space creates a bottleneck. Because the first classes "hog" the available mathematical territory, new classes introduced later are forced to compete for the same narrow regions.
This competition leads to feature overlap. Zhou et al. note that when new classes are added, they must overwrite existing knowledge to fit into the already-saturated space. To combat this, Shi and Ye propose a proactive approach: "squeezing" current class embeddings to reserve unallocated space for future categories. By treating the latent space as a finite resource that must be managed, models can accommodate new knowledge without causing collisions with established representations.
The Problem of Semantic Proximity
Forgetting is rarely random; it is often targeted. Le et al. and Huang et al. demonstrate that catastrophic forgetting is primarily driven by confusion between semantically or visually similar classes. When a model that already knows "car" is asked to learn "truck," the new features are naturally pulled toward the existing "car" representations.
This semantic interference makes it difficult to maintain distinct decision boundaries. Meng et al. identify this as "imbalanced forgetting," where classes most similar to the new data are forgotten significantly faster than unrelated ones. Phan et al. address this by using similarity-weighted knowledge distillation, which selectively reinforces the model's memory of old classes that are most visually similar to the new ones, preventing the boundaries from blurring.
Geometric Separation and Orthogonality
To prevent crowding, researchers have turned to the geometric properties of neural representations. A state known as "Neural Collapse"—where class representations are perfectly and equidistantly separated—is highly desirable but difficult to maintain during sequential updates. Seo et al. introduce a fixed geometric classifier to force new class features into a pre-defined, separated structure, reducing the bias that pulls novel features toward old ones.
Similarly, Deng suggests expanding the usable geometry by utilizing the full range of a hypersphere and explicitly reserving half of that space for future, unseen classes. The goal is to achieve "orthogonality"—ensuring that the vectors representing different tasks are as perpendicular as possible. As Ramasesh et al. show, larger models and those with more extensive pretraining naturally develop more orthogonal representations, making them inherently more resistant to the interference caused by feature overlap.
Disentangling Shared and Private Features
Another facet of the crowding problem is the entanglement of different types of knowledge. When a model learns a sequence of tasks, it often fails to distinguish between features that are universal (task-invariant) and those that are specific to a single category (task-specific). Ebrahimi et al. argue that without disentangling these components, updates for a new task will inevitably corrupt the shared features necessary for old ones.
By isolating these features—perhaps through architecture growth for private skills or specialized pathways as suggested by Zhu et al.—models can update their understanding of a specific domain without perturbing the broader feature space. This separation ensures that the "crowding" of new information is confined to its own neural pathway, leaving the representations of previous knowledge intact.