BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models

Kun Luo, Zheng Liu, Shitao Xiao, Kang LiuarXiv 2024 · DOI · arXiv

Introduction

Standard Retrieval-Augmented Generation (RAG) pipelines typically rely on a "chunk-and-retrieve" strategy. In this paradigm, massive documents are segmented into rigid, fixed-size text chunks (e.g., 200–500 words), which are then embedded and stored in a vector database. While this approach is computationally efficient, it introduces two fundamental flaws that limit the performance of Large Language Models (LLMs) on long-context tasks.

First, loss of coherence occurs when a sentence is isolated from its surrounding paragraph. Without the broader context, embeddings often fail to capture anaphoric references or logical transitions, leading to "thin" semantic representations. Second, information splitting occurs when critical data spans the boundary of two chunks. If a retriever only selects one of these chunks, the LLM receives an incomplete and potentially misleading fragment of information.

BGE Landmark Embedding (LE) proposes a "chunking-free" alternative. By using special landmark tokens to represent sentences within their original, coherent context and employing a position-aware training objective, this method significantly improves retrieval accuracy and downstream LLM performance without the need for expensive context window expansions.

Methodology

The Landmark Embedding (LE) framework replaces traditional chunking with a three-pronged technical approach designed to preserve document structure and semantic continuity.

1. Chunking-Free Architecture

Instead of breaking text apart, LE inserts a special <LMK> (landmark) token at the end of every sentence. The entire coherent sequence is then passed through an LLM-based encoder (such as LLaMA-2). The embedding of each <LMK> token serves as the representation for its preceding sentence. Because the token is processed alongside its neighbors within the LLM's attention mechanism, it "perceives" the full context. To handle extremely long documents, the system utilizes a sliding window, ensuring that even the edges of the window maintain local coherence.

2. Position-Aware Objective

In standard contrastive learning, all relevant sentences within a retrieved span are often treated as equally important. LE introduces a novel loss function that assigns higher weights to the "ultimate boundary"—the final sentence of a relevant information span. This encourages the model to act as a pointer, identifying exactly where a piece of information concludes. By retrieving the "front-k" sentences preceding that boundary, the system ensures that the LLM receives the full, uninterrupted context required to answer a query.

3. Multi-Stage Learning Pipeline

Training a model to understand long-context relationships requires a sophisticated data strategy. The authors employ a three-step pipeline:

  • Distant Supervision: Initial training on standard query-passage pairs (e.g., MS MARCO).
  • Weak Supervision: Training on "pseudo-long documents" created by merging answers from disparate queries to simulate complex retrieval environments.
  • Fine-Tuning: Utilizing high-quality synthetic data where an LLM (ChatGPT) generates questions based on specific, coherent spans within real-world Wikipedia articles.

Evaluation and Key Results

The researchers evaluated Landmark Embedding using LongBench, a comprehensive suite of six datasets including single-document QA (NarrativeQA, Qasper) and multi-document QA (HotpotQA, MuSiQue).

Performance Gains

On the LongBench tasks, Landmark Embedding improved the average score of LLaMA-2-7B by +8.8 points compared to a "no retrieval" baseline. It consistently outperformed state-of-the-art retrievers like E5-Mistral and BGE-large, even when those models were optimized with sentence-level chunking.

The "Needle in a Haystack" Test

In tests designed to find a specific fact hidden within 30,000 tokens, LE maintained a recall rate above 90% across the entire range. In contrast, traditional sliding window approaches often saw significant performance degradation as the size of the "haystack" increased.

Efficiency and Noise Reduction

One of the most striking findings was the efficiency gain for high-end models. Landmark Embedding improved the performance of ChatGPT-3.5-turbo (which possesses a 16K window) on tasks where information was scattered. LE achieved its best results using only ~2,100 input tokens, whereas ChatGPT without retrieval required over 15,000 tokens to achieve lower scores. This suggests that high-quality retrieval acts as a "noise reduction" layer, allowing LLMs to focus on relevant data rather than processing massive amounts of irrelevant text.

Discussion and Insights

The success of Landmark Embedding highlights several key insights for the future of RAG:

  • Contextualized Embeddings are Superior: A sentence embedding is significantly more discriminative when the encoder understands the narrative flow. By avoiding chunking, the model avoids the "semantic vacuum" of isolated snippets.
  • Retrieval as Pointing: By focusing on the end-point of a relevant span, the retriever functions more like a navigation tool for the document than a simple keyword matcher.
  • RAG vs. Long Context: Even as LLM context windows grow, retrieval remains essential for cost-efficiency and accuracy. LE provides a middle ground that leverages the strengths of both retrieval and long-context processing.

Limitations and Future Work

Despite its advantages, LE faces challenges regarding the cost of synthetic data generation, as the final training stage relies on expensive LLM APIs. Additionally, while the sliding window allows for theoretically infinite context, retrieval accuracy is still influenced by the window size used during training. Future research may explore integrating the "landmark" concept directly into the internal attention mechanisms of LLMs, potentially moving toward a more unified architecture for long-context understanding.

Reference: Kun, L., Liu, Z., Xiao, S., & Liu, K. (2024). BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models. arXiv:2402.11573.

Summary updated

Abstract

Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality of context extension. In this work, we proposeExtensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness. Extensible embedding stand as an enhancement of typical token embedding, which represents the information for an extensible scope of context instead of a single token. By leveraging such compact input units of higher information density, the LLM can access to a vast scope of context even with a small context window. Extensible embedding is systematically optimized in architecture and training method, which leads to multiple advantages. 1) High flexibility of context extension, which flexibly supports ad-hoc extension of diverse context lengths. 2) Strong sample efficiency of training, which enables the embedding model to be learned in a cost-effective way. 3) Superior compatibility with the existing LLMs, where the extensible embedding can be seamlessly introduced as a plug-in component. Comprehensive evaluations on long-context language modeling and understanding tasks verify extensible embedding as an effective, efficient, flexible, and compatible method to extend the LLM's context.
BibTeX
@misc{luo2024bge,
  author = {Kun Luo and Zheng Liu and Shitao Xiao and Kang Liu},
  title = {{BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models}},
  year = {2024},
  eprint = {2402.11573},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.11573},
}
Figure