This paper, recently presented at CIKM ’26, introduces a deceptively simple yet effective way to clean up the "noise" in text embeddings used for information retrieval. By prepending a few trainable "buffer" tokens to the input, the authors show we can significantly improve how models distinguish between relevant and irrelevant information.
TL;DR
The authors propose prepending trainable [DUMP] tokens to input text to act as semantic buffers that absorb non-semantic noise (like attention sinks) during encoding. This lightweight, plug-and-play method consistently improves retrieval performance across various encoder and decoder architectures on both standard and long-context benchmarks.
What Problem Does This Paper Address?
Modern retrieval systems rely on embedding models to compress variable-length text into a single, fixed-size vector. This creates a massive informational bottleneck. During this compression, "representational interference" occurs: the final vector often gets "polluted" by non-semantic information, such as positional artifacts or "attention sinks"—tokens that attract disproportionately high attention scores despite carrying little meaning (like stopwords or special tokens).
Standard fine-tuning tries to align queries and passages at a macro level but doesn't explicitly manage how this low-level noise is routed. The result is a diluted representation that limits the model's discriminative power.
Methods
The authors introduce a method called Dump-Token Conditioned Encoding.
- Prepending Tokens: They add $K$ indexed special tokens—
[DUMP1], ...,[DUMPk]—to the tokenizer vocabulary and prepend them to every input query and passage. - Trainable Buffers: These tokens are initialized in the model's embedding matrix and updated jointly with other parameters during retrieval fine-tuning.
- Standard Objective: Crucially, the model architecture, pooling strategy (e.g., using the
[CLS]token or the last token), and the contrastive loss function (InfoNCE) remain unchanged.
The [DUMP] tokens participate in the self-attention mechanism, interacting with regular text tokens. Because the final pooled vector relies on the attention distribution of the entire sequence, the training signal naturally encourages the [DUMP] tokens to "absorb" the structural or uninformative noise that would otherwise interfere with the semantic representation.
Datasets
- Training: A dataset of 110k instances, including 100k from RLHN (leveraging relabeled hard negatives) and 10k from MLDR (a long-document retrieval dataset).
- Evaluation:
- MTEB/BEIR subset: Includes Wikipedia, Natural Questions (NQ), HotpotQA, Quora, DBPedia, and MS MARCO.
- LongEmbed: Used to assess performance on long-context inputs where the compression challenge is even greater.
Key Insights and Ideas
- The "Semantic Buffer" Effect: The
[DUMP]tokens act as a dedicated structural pathway for noise. Analysis shows that attention from low-information tokens (like stopwords) is successfully redistributed away from the primary representation token (like[CLS]) and toward the[DUMP]tokens. - Activation Scaling: Without these tokens, "noise" tokens often exhibit unusually high activation norms (L2 norms). Adding
[DUMP]tokens reduces the L2 norm of these low-information tokens, leading to a "cleaner" internal state.
- Non-Monotonic Gains: Interestingly, adding more
[DUMP]tokens isn't always better. The optimal number of tokens ($K$) varies by model and dataset, suggesting it is a tunable hyperparameter.
Key Results
The method demonstrated robust improvements across six different embedding models, including both encoders (like GTE and NeoBERT) and decoders (like MobileLLM).
- Standard Retrieval: Achieved an average relative improvement of 2.46%. As seen in the table below, the "Ours" column consistently outperforms the "Base" across almost every category.
- Long-Context Retrieval: Achieved an average relative improvement of 2.41%. This is particularly significant as long-context inputs typically suffer more from representational interference.
- Architecture Agnostic: The gains were consistent regardless of whether the model used
[CLS]pooling (encoders) or last-token pooling (decoders).
Limitations and Future Work
- Hyperparameter Sensitivity: As shown in Figure 1, the best token count varies. For example, $K=2$ was best for
gte-mlmon average, while $K=4$ was better forMobileLLM. Finding the "sweet spot" for a specific architecture requires empirical testing. - Fixed vs. Dynamic Tokens: The current work uses a fixed number of prefix tokens. Future research could explore dynamic prefixing or different initialization strategies for the
[DUMP]tokens to further optimize their "noise-absorbing" capabilities.
Summary updated