Prototype Alignment for Evolving Feature Spaces

This direction is merged into another direction: representation drift, feature drift compensation. Its last version stays readable and citable here.

Version 1 of 1118 papers157 conceptsfrozen 2026-10-04

In modern machine learning, we often treat a model’s internal representation of data as a fixed map. We assume that once a model learns that "dogs" belong in a specific region of its feature space, that region remains the definitive home for dogs. However, in dynamic scenarios like continual learning or domain adaptation, this map is constantly being redrawn.

As a neural network learns new tasks or adapts to new environments, its weights change. This causes the underlying feature space to warp, shift, and rotate—a phenomenon known as feature drift or semantic drift. When the map changes but our "landmarks" (class prototypes or classifier weights) stay in their old positions, the model suffers from catastrophic forgetting.

Researchers have converged on a shared solution: instead of keeping landmarks static, we must dynamically compensate for drift to keep prototypes synchronized with the evolving feature space.

The Moving Target Problem

The core challenge, as identified by Bo-Cheng Qiu, Chia-Wen Lin, and Chih-Chung Hsu in CNN-Based Joint Clustering and Representation Learning with Feature Drift Compensation for Large-Scale Image Data, is that joint learning is inherently unstable. When you update a network to better cluster data, the very coordinates of that data change.

In incremental learning, this problem is even more acute. When a model trained on Task A is fine-tuned on Task B, the features for Task A classes "drift" away from their original centroids. Piotr Koniusz and Can Peng argue in Multivariate Prototype Representation for Domain-Generalized Incremental Learning that static representations of old classes become obsolete almost immediately as the backbone evolves.

Estimating Drift Without Old Data

The most elegant solutions to this problem are "exemplar-free"—they don't require storing old images, which is often impossible due to privacy or storage constraints. But if you don't have the old data, how do you know how much the feature space has shifted?

The field has developed several clever "proxy" mechanisms:

Geometric Realignment Strategies

Once the drift is estimated, the next question is how to apply the correction. Simple vector addition is often insufficient because the feature space doesn't just shift; it deforms.

In Prototypical quadruplet for few-shot class incremental learning, Biplab Banerjee and Subhasis Chaudhuri propose using statistical whitening and re-coloring. By adjusting the mean and variance of class distributions, they ensure historical anchors remain aligned with the current state of the feature extractor.

Other researchers use more complex transformations. Donghyeon Baek and Bumsub Ham utilize category-specific rotation matrices in ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation to preserve the internal cluster structures of old categories as weights shift. Similarly, Hiroshi Kera uses affine corrections in Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learning to realign representations after merging specialized model components.

Real-time and Test-time Adaptation

The most recent frontier in this research direction is moving drift compensation from the training phase to the inference phase.

Test-Time Classifier Adjustment (T3A), proposed by Yusuke Iwasawa and Yutaka Matsuo in Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization, demonstrates that we can adjust classifier weights on the fly. By calculating "pseudo-prototypes" from the incoming stream of unlabeled target data, the model can realign its decision boundaries to a new environment without any backpropagation.

This online philosophy is echoed in the work of Samantha Guerriero and Thomas Mensink. Their DeepNCM framework (DeepNCM: Deep Nearest Class Mean Classifiers) uses exponential moving averages to track drifting class centers continuously, ensuring that the "landmarks" are never more than a step behind the "map."

Why It Works: The Intuition of Stability

The success of these methods across image recognition, graph learning, and natural language processing suggests a fundamental principle: Classification is relative, not absolute.

A model doesn't need a static coordinate for "cat"; it needs a coordinate that is consistently positioned relative to the features the encoder is currently producing. By treating prototypes as dynamic entities that can be transported (Pascal Mettes, Universal Prototype Transport for Zero-Shot Action Recognition and Localization) or predicted (Daniel Marczak, Generalized Continual Category Discovery), we decouple the stability of the classifier from the plasticity of the feature extractor. This allows models to learn and adapt indefinitely without losing their grip on the past.

Papers in this version

Showing 60 of 118 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_prototype_alignment_for_evolving_feature_3c22edf9,
  title = {Prototype Alignment for Evolving Feature Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prototype-alignment-for-evolving-feature-spaces/rdr_3c22edf9e7f44fec968cc8d2f8082382},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_prototype_alignment_for_evolving_feature_3c22edf9_v1,
  title = {Prototype Alignment for Evolving Feature Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prototype-alignment-for-evolving-feature-spaces/rdr_3c22edf9e7f44fec968cc8d2f8082382/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}