Backfill-free representation learning

22 concepts14 papers1 proposal

Imagine running a large-scale visual search engine, face recognition system, or document retrieval platform containing billions of items. Each item is stored as a dense vector embedding generated by a deep neural network. Over time, you design a new model architecture or train on a larger dataset, resulting in a significantly more accurate model.

In a traditional deployment pipeline, upgrading to this new model introduces a massive logistical bottleneck known as backfilling. Because the new model's embedding space is completely different from the old one, you cannot directly compare a new query vector against the existing database. To deploy the upgrade, you must re-encode every single item in your multi-billion-item gallery—a process that is computationally prohibitive, environmentally expensive, and introduces significant system downtime.

Backward-compatible representation learning solves this dilemma. The core idea is to train an upgraded model such that its new embeddings can be directly compared against the legacy embeddings already stored in the database. This allows for instant system upgrades and "hot-refreshes" without requiring any database re-indexing.

Aligning with Legacy Decision Boundaries

The most intuitive way to make two embedding models compatible is to force the new model to respect the structure of the old one. However, if you force the new feature vectors to be identical to the old ones, you severely limit the new model's capacity to learn better, more discriminative representations.

The foundational insight to solve this, introduced by Yantao Shen et al. in Towards Backward-Compatible Representation Learning, is that the new model does not need to duplicate the exact coordinates of the old features. Instead, it only needs to respect the decision boundaries established by the previous system. By training the new embedding model using a loss term evaluated against the fixed classifier head of the legacy model, the new features are guaranteed to land on the correct side of the old model's decision boundaries.

Legacy Space:   [ Class A Boundary | Class B Boundary ]
                                       ^
New Model maps features here ----------| (Respects boundary, but can optimize internal spacing)

To give the new model even more flexibility, Qiang Meng et al. proposed a point-to-set alignment strategy in Learning Compatible Embeddings. Rather than constraining individual data points, the new model aligns its updated representations with the class centers and angular boundaries of the legacy model. This point-to-set approach ensures that new embeddings remain within the correct semantic neighborhood of legacy data while freeing the backbone to optimize its internal feature clustering. Building on this concept, Furong Xu et al. demonstrated in Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval that incorporating adversarial learning and elastic boundary constraints can further refine these margins, maximizing both cross-model compatibility and standalone retrieval accuracy.

Preserving Relative Similarity Structures

An alternative way to achieve compatibility is to focus on the geometric relationships between items rather than their absolute classification boundaries. If two images are highly similar in the legacy space, they should maintain a similar relative closeness when processed by the updated model.

As shown by Nicu Sebe et al. in Anti-Forgetting Adaptation for Unsupervised Person Re-Identification, maintaining the relative similarity structure across model updates unifies the goals of preventing catastrophic forgetting and ensuring feature stability. By enforcing consistency between the similarity relationships of old samples as calculated by the current model and a frozen previous version, the model preserves the underlying topology of the feature space.

This geometric anchoring allows multi-generation embeddings to coexist seamlessly in the same index. For instance, the frameworks proposed by Jun-Cheng Chen et al. in Continual Learning for Visual Search with Backward Consistent Feature Embedding and Yukang Gan et al. in Binary Embedding-based Retrieval at Tencent utilize contrastive learning objectives to anchor new model outputs directly to the coordinate systems of previous versions. This ensures that new queries can be successfully executed against legacy document or image stores without re-extraction.

Post-Training Transformations and Dimension Isolation

In many production environments, modifying the core training loop of a massive foundation model to enforce compatibility constraints is impractical. In these scenarios, researchers use lightweight transformation layers or structural embedding designs to bridge disparate spaces during or after training.

One approach is to learn a mapping between the old and new spaces post-hoc. In Learning Backward Compatible Embeddings, Weihua Hu et al. introduced a framework that trains a lightweight transformation layer to map new high-dimensional vectors back into the legacy vector space. This allows upstream providers to upgrade architectures freely while downstream consumers continue using models trained on old embeddings. Similarly, in cross-modal setups like vision-language models, Young Kyun Jang et al. showed in Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models that a text-only pre-trained projection module can successfully align new model embeddings with old ones. To handle bidirectional lookups (matching new queries to old galleries, and old queries to new galleries), Simone Ricci et al. proposed a dual-adapter system in $\lambda$-Orthogonality Regularization for Compatible Representation Learning that simultaneously learns forward and backward transformations between legacy and updated latent spaces.

[New Query] ---> (New Model) ---> [New Embedding] ---> (Transformation Layer) ---> [Legacy-Compatible Vector]
                                                                                            |
                                                                                    Matches directly against
                                                                                            |
                                                                                            v
                                                                                    [Legacy Gallery Index]

However, forcing a brand-new model's representation entirely into an older, lower-capacity space can bottleneck its performance. To bypass this trade-off, Yifei Zhou et al. proposed an elegant basis transformation technique in BT²: Backward-compatible Training with Basis Transformation.

Instead of restricting the new model's size or layout, the upgraded model outputs an expanded embedding where entirely new, highly discriminative features are isolated in extra dimensions.

$$\mathbf{z}_{\text{new}} = [\mathbf{z}_{\text{compatible}} \mid \mathbf{z}_{\text{novel}}]$$

When performing cross-model retrieval against the legacy database, these extra dimensions ($\mathbf{z}_{\text{novel}}$) are simply truncated, leaving an information-preserving representation that perfectly aligns with the legacy dimensionality. When performing new-to-new retrieval, the system utilizes the full extended embedding, preserving the maximum accuracy gains of the upgraded model.

Key Foundational Work

Towards Backward-Compatible Representation Learning (2020) — contributed 4 concept elements to this direction.
Learning Compatible Embeddings (2021) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_backward_compatible_representation_learn_bd312515,
  title = {backward-compatible representation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backward-compatible-representation-learning/rdr_bd312515d949443a9a7d4736f669452f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_backward_compatible_representation_learn_bd312515_v1,
  title = {backward-compatible representation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backward-compatible-representation-learning/rdr_bd312515d949443a9a7d4736f669452f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake