Some search systems now convert a dense text embedding into a short list of human-readable "concepts" and store documents in an ordinary inverted index keyed by those concepts. A dense embedding is a vector with hundreds of dimensions, all active at once, that no one can read. An inverted index is the classic keyword-search data structure: each key maps to the documents that contain it. The conversion is done by a sparse autoencoder, a small network that rewrites each dense vector as a few active units, each unit standing for one nameable idea. These systems match on meaning rather than surface words, so they get past the vocabulary mismatch problem, where a query and its answer share no words, while staying cheap and inspectable. They are also frozen in time. When the corpus drifts toward new topics and jargon, the concept dictionary learned on old data stops covering what the new documents are about. The standard remedy, retraining the autoencoder, silently changes what every existing unit means. The stored index, and every human label attached to its units, then refers to concepts that no longer exist.
We propose to grow the concept dictionary in an append-only way. Old concept units stay frozen, and old postings are never rewritten. New units are added in a separate block that is trained only on what the old dictionary fails to explain in new data. Each block selects its active units independently of the others. Under this design, a query's old-block concepts are exactly what they would have been before the update, so retrieval of old documents is provably unchanged, with zero re-indexing. All forgetting is pushed into one legitimate place: new documents competing for rank. The research question is whether this constrained growth still tracks the frozen dense encoder on new domains, and whether it does so at a small fraction of the cost of retraining and re-indexing.
Core Claim
The closest work is Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval. It trains a BatchTopK autoencoder, one that keeps only the top-scoring units across a batch, on DPR embeddings, and indexes the resulting concepts with IDF weighting. The system, called CL-SR, reaches MRR@10 of 0.368 on MS MARCO Dev with a 0.57 GB index, and it degrades less than lexical search on queries with no word overlap. The companion paper Interpret and Control Dense Retrieval with Sparse Latent Features shows sparse codes over BGE retaining about 92% of dense MRR (0.3306 vs. 0.3605) and supports steering retrieval through labeled units. Both papers are static. Both also note that the learned features are incomplete and depend on the training data. That observation motivates this proposal: it predicts exactly the coverage loss under drift that we target.
Our claim has two parts. The first is a failure-mode claim. Naively updating such an autoencoder causes a failure we call code competition: under a single shared top-k budget, newly trained units steal active slots from old units on old-domain queries. Old queries then stop matching their old postings even when the dense encoder never changed. The second is a design claim. Separating the per-block selection budgets, and gating new units so they fire only when old-block reconstruction error is high, removes code competition by construction and recovers most of the new-domain gap to dense retrieval.
Formally, let $x$ be a frozen dense embedding and $D_{<t}$ the frozen decoder of all blocks before session $t$. The old code $z_{<t}(x)$ is computed exactly as before. The new block's encoder $E_t$ and decoder $D_t$ are trained only on new-session embeddings, using the residual:
$$\min_{E_t, D_t} \; \mathbb{E}_{x \sim \mathcal{S}_t} \left\| x - D_{<t} z_{<t}(x) - D_t \, \mathrm{TopK}_{k_t}\!\big(g(x)\, E_t x\big) \right\|^2$$
Here $\mathcal{S}_t$ is the session-$t$ document stream and $k_t$ is the new block's own active-unit budget. The gate $g(x) \in [0,1]$ opens only when the old-block residual norm exceeds a threshold calibrated on old data. The retrieval score is a sum of per-block dot products. Old documents have no postings in new blocks, so the score between an old query and an old document is identical before and after every update. IDF is applied on the query side, so recomputing it as the corpus grows touches no postings.
Research Program
The work has three parts. First, we measure the failure. We train a BatchTopK autoencoder on MS MARCO embeddings from a frozen encoder, then add domains one at a time. After each step we compare three conditions: no update, fine-tuning all units, and retraining from scratch. For each, we measure unit identity churn, which is the fraction of old units whose best-matched successor has decoder cosine similarity below 0.8. We also measure how much old-domain nDCG@10 falls when old postings are kept but queries are encoded with the updated model.
Second, we build block-wise growth and its gate. We study how gate threshold, $k_t$, and block size trade new-domain accuracy against index growth. We use a residual-energy trigger to decide whether a session needs a new block at all.
Third, we check that the new units are still interpretable. We reuse the evaluation of the CL-SR paper, where annotators identified passages from LLM-written unit descriptions with 0.943 accuracy, and add a word-intrusion style test in the spirit of SPINE. Residual-trained units could plausibly turn into uninterpretable "error-correction" directions. That is a real risk, and we report it either way.
This differs from the other relevant intersections as follows.
CREAM fights forgetting over streaming corpora with self-supervised soft memory over token-level dense representations. It never touches sparse inverted indexes.
AURORA routes between dense, sparse and generative retrievers under lexical drift, treating sparsity as an external modality. It simulates drift through synthetic token replacement on NFCorpus.
Additive Parameter Decomposition supplies the structural idea we reuse: a frozen shared base plus sparse task-specific additions. Its constraints shape our design. APD needs task identity at inference, whereas our gate replaces task identity with a reconstruction-error signal. APD also works on weights for image classification, whereas we apply the base-plus-residual structure to units of meaning in a retrieval index.
Simple Domain Adaptation for Sparse Retrievers adapts SPLADE's lower layers to one new domain, gaining 0.7 to 1.4 nDCG@10. It never measures what the source domain loses, which is our continual SPLADE comparison.
Baselines
The do-nothing reference keeps the MS MARCO-trained autoencoder and index frozen and simply encodes new documents with it. This is CL-SR as published, run on our session stream with the same encoder. Old-domain accuracy is preserved trivially, so the only question is how much new-domain accuracy it loses. Our method must beat it on new-domain nDCG@10 while tying it on old domains.
The upper bound retrains the autoencoder on all data seen so far and re-indexes the full corpus. It gets the same autoencoder capacity and training steps as our cumulative blocks. Beating it is not required. Success means coming within one nDCG@10 point of it on average while re-encoding zero old documents.
The frozen dense encoder with approximate nearest-neighbor search is the accuracy ceiling for any autoencoder-on-top approach. Neither our method nor the upper bound can exceed it. We report our fraction of dense accuracy per domain, alongside the 92% retention figure from the Interpret and Control paper.
The strongest trained competitor is continual SPLADE. We start from DistilSPLADE-max from SPLADE v2, which reports 0.368 MRR@10 on MS MARCO and an average of 0.500 nDCG@10 over 13 BEIR datasets. We adapt it sequentially with the lower-layer MLM recipe of Vast et al. and give it a replay buffer of the same byte size as our added index blocks. We run it in two variants. One re-encodes old documents after each step, charging that cost. The other keeps stale postings, so we can observe forgetting in the drift of learned expansions. Our method beats it if it matches or exceeds its average nDCG@10 across sessions at lower re-encoding cost, or matches that cost with less backward forgetting.
The key ablation is shared-budget growth: the same new units, but competing with old units in one top-k. It isolates code competition. A second ablation removes the gate. BM25 is reported as the lexical floor.
Why not just…?
Why not just use the frozen dense encoder, which never forgets? On accuracy this is a real threat, and we treat it as the ceiling rather than a competitor. Its limits lie elsewhere. It produces no inspectable units, it cannot support the signed negation and set operations that lexically grounded sparse indexes allow in Constructing Set-Compositional and Negated Representations for First-Stage Ranking, and it needs vector-index infrastructure. If our method cannot stay close to dense on new domains, the case for concept indexes under drift weakens, and we will say so.
Why not just retrain the autoencoder and re-index? For small corpora this may simply be fine. We will report the crossover corpus size at which re-indexing cost dominates. Retraining still churns unit identities, however, which throws away every human- or LLM-written description and every steering rule built on the old units. Our churn measurement quantifies this loss directly.
Why not just run continual SPLADE, whose dimensions are fixed vocabulary tokens? Token dimensions do not change meaning, but the learned expansion weights do. Documents encoded before an update then disagree with queries encoded after it, unless everything is re-encoded. New jargon also fragments into subword pieces. This is the baseline most likely to win on raw accuracy, and the matched-budget comparison above settles it.
Why not just use a replay buffer? CREAM argues that labeled replay is unavailable in unlabeled streams. More to the point, replay reduces forgetting in parameters but does nothing about stale postings. We include replay inside the SPLADE baseline rather than as a separate method.
Proposed Experiments
The stream starts with MS MARCO. It then adds TREC-COVID, NFCorpus, FiQA-2018 and TREC-NEWS as separate sessions, drawing on BEIR (BEIR). A second stream uses LoTTE's topic splits to follow the 10-session protocol of CREAM. We use both DPR, for comparability with CL-SR, and BGE-base-en-v1.5 as frozen encoders. After every session we evaluate every prior domain on the growing union corpus.
We report per-domain nDCG@10, MRR@10 on MS MARCO Dev, and backward transfer. Backward transfer is the change in each domain's score after later sessions. Under our design it should be exactly zero on old-only corpora and non-positive only because of new documents entering the union. We also report the number of re-encoded documents, index size, query FLOPs, and the interpretability scores described above. Old-domain accuracy is measured immediately after each switch, so any stability-gap-style transient of the kind reported in Overcoming the Stability Gap in Continual Learning would be visible. For the baselines we expect one there; for ours we predict none.
For reference numbers, the strongest static results in the file are CL-SR and DistilSPLADE-max, both at 0.368 MRR@10 on MS MARCO Dev, and DistilSPLADE-max at 0.500 average nDCG@10 on BEIR. No reported result was found on Lacuna for CL-SR's nDCG@10 on TREC-COVID or NFCorpus, and none was found for any autoencoder-based retriever under sequential domain updates. We will establish both.
Three outcomes decide the proposal. First, shared-budget growth must measurably degrade old-domain retrieval while block-wise growth does not. If shared-budget growth shows no code competition, the gate and blocks are unnecessary, and plain fine-tuning with frozen old units suffices. Second, block-wise growth must close at least half of the new-domain gap between the frozen autoencoder and frozen dense retrieval. If it does not, residual units are not capturing the missing concepts. Third, new units must pass description-based identification at rates comparable to old units. If they fail, growth preserves the index but sacrifices the interpretability that justified a concept index in the first place.
References
- Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval
- Interpret and Control Dense Retrieval with Sparse Latent Features
- SPINE: SParse Interpretable Neural Embeddings
- CREAM: Continual Retrieval on Dynamic Streaming Corpora with Adaptive Soft Memory
- AURORA: Neuro-Symbolic Continual Indexing for Evolving RAG Systems
- Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
- Simple Domain Adaptation for Sparse Retrievers
- SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
- Constructing Set-Compositional and Negated Representations for First-Stage Ranking
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Overcoming the Stability Gap in Continual Learning