TL;DR
This paper introduces "Embedding-Free RAG," a novel approach to Retrieval-Augmented Generation (RAG) that eliminates the need for traditional embedding models. By leveraging the reasoning capabilities of Large Language Models (LLMs) directly for retrieval, it offers a simpler, more adaptable RAG pipeline that achieves state-of-the-art performance on challenging legal and financial question-answering tasks.
What Problem Does This Paper Address?
Current Retrieval-Augmented Generation (RAG) systems, while effective at grounding LLM responses with external knowledge, are notoriously complex to deploy. They typically require significant expertise in Natural Language Processing (NLP) for tasks like fine-tuning embedding models for specific domains, optimizing text chunking strategies, and hyperparameter tuning. This complexity makes it difficult to implement RAG for real-world applications, especially in specialized fields or when dealing with user-provided documents on-the-fly (the "Chat with a Document" scenario). Furthermore, traditional RAG struggles with the "lost in the middle" phenomenon in long-context LLMs, where information in the middle of a long input is often ignored.
Methods
The core innovation of Embedding-Free RAG is its elimination of embedding models for retrieval. Instead, it uses an LLM-driven approach in two key steps:
- Anchor Creation: The LLM is prompted to generate "reference quotations" – exact text spans from the user-provided document that are relevant to the query. To handle potential variations in LLM output (e.g., slight differences in wording or spacing), the paper uses the Levenshtein distance to map these quotations back to the most similar sentence in the original document. These identified sentences serve as "anchors."
- Chunk Construction: To provide sufficient context for the LLM to generate an answer, the anchors are expanded into "chunks." This is done by taking a fixed number of sentences before and after each anchor. Overlapping chunks are then merged to create contiguous text segments.
Finally, these generated chunks are passed to the LLM for answer generation, mimicking the final stage of traditional RAG. The authors also propose optimizations like parallelized document processing to handle very large documents and the use of different LLMs for the retrieval and generation stages.
Datasets
The paper evaluates Embedding-Free RAG on two challenging benchmarks:
- LegalBench-RAG: This benchmark focuses on question answering for legal documents and includes four datasets: PrivacyQA, CUAD, MAUD, and ContractNLI. It's designed to test RAG's ability to extract concise, highly relevant text segments.
- FinanceBench: This benchmark evaluates LLMs on financial question answering using data from publicly traded US companies and their filings. It includes questions requiring domain-specific knowledge and numerical reasoning.
Key Insights and Ideas
- LLMs as Retrievers: The most significant insight is that LLMs themselves can be effectively used for the retrieval stage, bypassing the need for specialized embedding models. This simplifies the RAG pipeline considerably.
- "Chat with a Document" Suitability: Embedding-Free RAG is particularly well-suited for scenarios where documents are provided at inference time, as it avoids the need for pre-computation or offline indexing.
- Improved Trustworthiness and Transparency: By surfacing the exact text used to generate answers, Embedding-Free RAG enhances transparency and trustworthiness, which is crucial in sensitive domains.
- Mitigating Long-Context LLM Issues: The anchor-based retrieval helps address the "lost in the middle" problem by focusing on relevant snippets rather than processing the entire long document at once.
- Domain Agnosticism: The model-agnostic nature of the approach means it can be applied across various domains without extensive domain-specific tuning.
Key Results
- Superior Performance: Embedding-Free RAG significantly outperforms existing state-of-the-art RAG methods on both LegalBench-RAG and FinanceBench.
- On LegalBench-RAG, it achieved an average improvement of 2.6x in F1 score, with up to a 4.6x improvement on the MAUD dataset.
- On FinanceBench, it achieved 83% accuracy, outperforming traditional RAG (50%) and long-context LLMs (79%), and coming close to the "oracle" setting (85%) which represents perfect retrieval.
- Efficiency: The approach minimizes setup costs and enables rapid deployment, especially in specialized or low-resource domains, by removing the need for expensive domain-specific datasets or fine-tuning.
- LLM Flexibility: The performance of Embedding-Free RAG is robust across different LLMs used for quotation generation, indicating the framework's generalizability.
Limitations and Future Work
- Not for All Tasks: Embedding-Free RAG is not ideal for summarization tasks where reference quoting is not well-defined or for large-scale applications with fixed document sets where traditional RAG might be more cost-effective long-term.
- Long-Context LLM Inference: For less challenging, non-technical documents, long-context LLM inference alone might suffice, making the added complexity of RAG unnecessary.
- Structured Document Challenges: While the paper explores structured documents (like PDFs), using OCR for text extraction surprisingly led to worse performance on tabular data, suggesting that markdown might not be the optimal representation for LLMs. Future work could explore better ways to represent tabular data.
- Further Optimizations: The authors suggest exploring more advanced chunking methods when additional structural information is available and further investigating the impact of different LLMs for retrieval and generation.
Summary updated