Managing Semantic Drift in Evolving Feature Spaces

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2583 papers686 conceptsfrozen 2026-10-04

In the standard paradigm of machine learning, we treat a model’s latent space as a static coordinate system. Once a model learns to map an image of a "cat" to a specific vector, we expect that location to remain meaningful. However, in continual learning—where models must learn new tasks sequentially—this coordinate system is under constant renovation. As the model’s backbone updates to accommodate new information, the internal "map" shifts, rotates, and stretches.

This phenomenon, known as semantic drift, is the hidden engine behind catastrophic forgetting. When the feature extractor evolves but the classification head remains static, the model’s "eyes" and its "brain" fall out of alignment.

The Moving Target Problem

The core challenge of incremental learning is that the feature extractor (the backbone) and the classifier (the head) are often updated at different rates or under different constraints. Daniel Vidal and colleagues in their work on Prototype Augmented Hypernetworks point out that as the backbone's internal representations shift, a "feature gap" opens up. The static decision boundaries established for previous tasks no longer contain the drifted representations of those same classes.

Imagine a librarian who reorganizes the shelves (the backbone) every time a new genre of books arrives. If the index cards (the classifier) aren't updated to reflect the new shelf locations, the librarian will "forget" where the old books are, even though the books are still in the building. As Heechul Jung noted in early work on Less-forgetful Learning, without constraints to maintain spatial consistency, the original decision boundaries become invalid almost immediately upon fine-tuning.

Geometry of the Drift

Semantic drift isn't just a random jitter; it often involves a systematic transformation of the latent space geometry. Haizhou Shi and Zixuan Ni categorize this as "classifier deviation" in their C4IL framework, where the evolved representation space simply stops matching the static decision boundaries.

This drift can be visualized in three ways:

  1. Translation: The entire cluster for a class moves to a new neighborhood in the latent space, as explored in ZSTCI by Kun Wei and Cheng Deng.
  2. Rotation and Scaling: The relative angles between class vectors change, breaking backward compatibility with older model versions. Niccolò Biondi and Federico Pernici address this in CoReS by using fixed geometric "polytopes" to force the feature space to remain stationary.
  3. Entanglement: As new classes are added, the boundaries of old classes blur. Fan Lyu describes this in MDMT Rehearsal as a partitioning problem, where new information "crowds out" the latent space reserved for old tasks.

Anchoring the Latent Space

To combat this, researchers have moved toward "anchoring" the feature space to something that doesn't change.

One powerful anchor is Language. Because the relationship between the words "dog" and "cat" is semantically stable in human language, it can serve as a fixed reference point for a shifting visual model. Tao Zhou and Yi Zhou demonstrate in Language as an Anchor that using language-derived geometry prevents the relative relationships between classes from shifting across domains. Similarly, Donghao Sun uses Large Multimodal Models in Dual-Space Semantic Synergy to provide a "semantic stability" that visual-only systems lack.

Another approach is Prototypical Calibration. Instead of assuming the feature space is static, methods like TRIPS by Piotr Koniusz treat class representations as evolving distributions. By modeling the drift itself, these systems can "re-align" old knowledge with the new backbone. Sathursan Kanagarajah takes this further in SATHUR by using a "hallucinator" to generate synthetic features that reflect how old data would look in the new, drifted feature space.

Why It Matters for Interpretability

The problem of semantic drift isn't limited to classification; it also plagues our ability to understand how models learn. Ryosuke Takahashi and Benjamin Heinzerling note in their work on TopK Language Models that tracking the "birth" of a concept during training is nearly impossible because features don't have a persistent identity. As the model trains, a feature representing "justice" might migrate across different neurons or layers.

By solving semantic drift, we aren't just preventing forgetting; we are building models with a persistent identity, where concepts have a stable home in the latent space from the first epoch to the last.

Papers in this version

Showing 60 of 583 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 575 papers
  2. Version 1 · 2026-10-04 · 583 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_representation_drift_d5e0f432,
  title = {representation drift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representation-drift/rdr_d5e0f43203ba4b63988a2932a3813679},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_managing_semantic_drift_in_evolving_feat_d5e0f432_v1,
  title = {Managing Semantic Drift in Evolving Feature Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/managing-semantic-drift-in-evolving-feature-spaces/rdr_d5e0f43203ba4b63988a2932a3813679/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}