Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens

Seongtae Hong, Youngjoon Jang, Jungseob Lee, Chanjun Park, Heuiseok LimarXiv 2026 · arXiv

This paper, recently presented at CIKM ’26, introduces a deceptively simple yet effective way to clean up the "noise" in text embeddings used for information retrieval. By prepending a few trainable "buffer" tokens to the input, the authors show we can significantly improve how models distinguish between relevant and irrelevant information.

TL;DR

The authors propose prepending trainable [DUMP] tokens to input text to act as semantic buffers that absorb non-semantic noise (like attention sinks) during encoding. This lightweight, plug-and-play method consistently improves retrieval performance across various encoder and decoder architectures on both standard and long-context benchmarks.

What Problem Does This Paper Address?

Modern retrieval systems rely on embedding models to compress variable-length text into a single, fixed-size vector. This creates a massive informational bottleneck. During this compression, "representational interference" occurs: the final vector often gets "polluted" by non-semantic information, such as positional artifacts or "attention sinks"—tokens that attract disproportionately high attention scores despite carrying little meaning (like stopwords or special tokens).

Standard fine-tuning tries to align queries and passages at a macro level but doesn't explicitly manage how this low-level noise is routed. The result is a diluted representation that limits the model's discriminative power.

Methods

The authors introduce a method called Dump-Token Conditioned Encoding.

  1. Prepending Tokens: They add $K$ indexed special tokens—[DUMP1], ..., [DUMPk]—to the tokenizer vocabulary and prepend them to every input query and passage.
  2. Trainable Buffers: These tokens are initialized in the model's embedding matrix and updated jointly with other parameters during retrieval fine-tuning.
  3. Standard Objective: Crucially, the model architecture, pooling strategy (e.g., using the [CLS] token or the last token), and the contrastive loss function (InfoNCE) remain unchanged.

The [DUMP] tokens participate in the self-attention mechanism, interacting with regular text tokens. Because the final pooled vector relies on the attention distribution of the entire sequence, the training signal naturally encourages the [DUMP] tokens to "absorb" the structural or uninformative noise that would otherwise interfere with the semantic representation.

Datasets

  • Training: A dataset of 110k instances, including 100k from RLHN (leveraging relabeled hard negatives) and 10k from MLDR (a long-document retrieval dataset).
  • Evaluation:
    • MTEB/BEIR subset: Includes Wikipedia, Natural Questions (NQ), HotpotQA, Quora, DBPedia, and MS MARCO.
  • LongEmbed: Used to assess performance on long-context inputs where the compression challenge is even greater.

Key Insights and Ideas

  • The "Semantic Buffer" Effect: The [DUMP] tokens act as a dedicated structural pathway for noise. Analysis shows that attention from low-information tokens (like stopwords) is successfully redistributed away from the primary representation token (like [CLS]) and toward the [DUMP] tokens.
  • Activation Scaling: Without these tokens, "noise" tokens often exhibit unusually high activation norms (L2 norms). Adding [DUMP] tokens reduces the L2 norm of these low-information tokens, leading to a "cleaner" internal state.
Figure 2: Token-level analysis of the [DUMP] mechanism using gte-en-mlm-base. Panel (a) compares token-group activation scales measured by the mean L2 norm. Panel (b) visualizes the attention distribution from low-information tokens to [CLS] and D1–D4, with D1–D4 ordered by prefix position.
  • Non-Monotonic Gains: Interestingly, adding more [DUMP] tokens isn't always better. The optimal number of tokens ($K$) varies by model and dataset, suggesting it is a tunable hyperparameter.

Key Results

The method demonstrated robust improvements across six different embedding models, including both encoders (like GTE and NeoBERT) and decoders (like MobileLLM).

  • Standard Retrieval: Achieved an average relative improvement of 2.46%. As seen in the table below, the "Ours" column consistently outperforms the "Base" across almost every category.
  • Long-Context Retrieval: Achieved an average relative improvement of 2.41%. This is particularly significant as long-context inputs typically suffer more from representational interference.
  • Architecture Agnostic: The gains were consistent regardless of whether the model used [CLS] pooling (encoders) or last-token pooling (decoders).
Table 1: Retrieval results on standard benchmark datasets. The table reports nDCG@10 and compares fine-tuned baseline models with their counterparts trained using [DUMP] tokens across encoder and decoder architectures.

Limitations and Future Work

  • Hyperparameter Sensitivity: As shown in Figure 1, the best token count varies. For example, $K=2$ was best for gte-mlm on average, while $K=4$ was better for MobileLLM. Finding the "sweet spot" for a specific architecture requires empirical testing.
  • Fixed vs. Dynamic Tokens: The current work uses a fixed number of prefix tokens. Future research could explore dynamic prefixing or different initialization strategies for the [DUMP] tokens to further optimize their "noise-absorbing" capabilities.
Figure 1: Effect of the number of [DUMP] tokens on retrieval performance. We compare one, two, and four [DUMP] tokens with gte-mlm-base as the encoder and MobileLLM-R1-140M as the decoder. Bars indicate performance changes relative to the corresponding baseline without [DUMP] tokens.

Summary updated

Abstract

Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive retrieval objective. By introducing trainable prefix positions that interact with regular tokens during encoding, [DUMP] tokens help reduce the influence of residual or uninformative token-level signals on the final retrieval representation. Experiments on standard retrieval and long-context benchmarks show that our method consistently improves over finetuned baselines across encoder and decoder embedding models. Further analysis shows that [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens. These results suggest that trainable prefix tokens provide a lightweight way to improve retrieval representations.
BibTeX
@misc{hong2026reducing,
  author = {Seongtae Hong and Youngjoon Jang and Jungseob Lee and Chanjun Park and Heuiseok Lim},
  title = {{Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens}},
  year = {2026},
  eprint = {2611.07298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07298},
}
Figure