TL;DR
This paper introduces BRIGHT+, a significantly cleaned and re-chunked version of the BRIGHT benchmark, designed to better evaluate Retrieval-Augmented Generation (RAG) systems. They developed MARCUS, a multi-agent LLM pipeline, to systematically remove noise and improve semantic coherence, leading to substantial improvements in retrieval accuracy and reasoning performance.
What Problem Does This Paper Address?
Retrieval-Augmented Generation (RAG) systems are powerful tools for question answering and text generation, but their performance heavily relies on the quality of the retrieval corpus. The BRIGHT benchmark, while designed for complex multi-hop reasoning, suffers from common issues found in web-scraped data: structural noise (like boilerplate text) and semantic discontinuity (where related information is fragmented across chunks). These flaws make it harder for RAG models to accurately retrieve relevant information and perform complex reasoning, essentially forcing them to sift through "shuffled books." The authors found these issues are particularly concentrated in StackExchange-derived subdomains within BRIGHT.
Methods
The core of this work is the MARCUS pipeline, a novel LLM-driven system designed for cleaning and segmenting retrieval corpora. MARCUS operates in three stages:
- Aggregate: Documents are first classified as "Gold" (containing answer-relevant information) or "Irrelevant." Gold documents are further split into "Gold Parts" (answer-bearing spans) and "Irrelevant Parts" (noisy segments).
- Clean: Two specialized LLM agents handle noise removal:
- SafeClean: Focuses on Gold Parts, preserving semantic integrity while removing interface-level clutter (headers, sidebars, etc.) without paraphrasing.
- FastClean: Aggressively removes noise and boilerplate from Irrelevant Parts and entire Irrelevant Documents, prioritizing speed and thoroughness.
- Split: A Splitter agent uses LLMs to segment the cleaned text into semantically coherent, self-contained chunks. This stage aims to maximize intra-chunk coherence and minimize the number of chunks, introducing boundaries only at significant topic shifts.
Datasets
- BRIGHT: The original benchmark dataset used as the basis for cleaning and improvement.
- BRIGHT+: The newly created, cleaned, and re-chunked version of the BRIGHT dataset.
Key Insights and Ideas
- Web-scraped data is inherently noisy: The paper clearly demonstrates that even well-intentioned benchmarks like BRIGHT can be significantly hampered by common web artifacts, impacting downstream RAG performance.
- LLMs are powerful for corpus cleaning: MARCUS showcases the effectiveness of LLMs not just for generation but also for sophisticated data preprocessing tasks like noise removal and semantic segmentation.
- Semantic chunking is crucial for RAG: The ablation studies strongly suggest that the semantic segmentation (splitting) stage of MARCUS is the primary driver of performance improvements, even more so than the noise-cleaning stage. This highlights the importance of well-formed, coherent chunks for effective retrieval.
- Reranking benefits from strong LLMs: While lightweight rerankers offer limited gains, powerful LLMs like GPT-3.5-Turbo can significantly boost performance, especially when dealing with noisy candidate sets or requiring deeper semantic understanding.
Key Results
- Substantial chunk reduction: MARCUS reduced the total chunk count in BRIGHT by 97%, retaining only 3.56% of the original segments, resulting in longer, more coherent chunks.
- Improved retrieval accuracy: BRIGHT+ led to significant improvements in retrieval performance (nDCG@10) across various retrievers, with dense models showing particularly marked gains. For instance, QWEN2 saw an average improvement of 19.8% on BRIGHT+.
- Enhanced RAG performance: End-to-end RAG evaluations showed that MARCUS-S (splitting only) accounted for roughly 83% of the total gain achieved by the full MARCUS pipeline, with the full MARCUS pipeline securing the highest scores.
- Reasoning query benefits: LLM-generated reasoning-style queries led to significant gains in nDCG@10, especially for weaker retrievers like BM25 and BGE.
- Reranking effectiveness: GPT-3.5-Turbo reranking at k=10 improved performance by over 5 absolute points for strong retrievers like GritLM and QWEN2 compared to MiniLM.
Limitations and Future Work
- Computational Cost: The reliance on large LLMs for MARCUS incurs high computational costs, making scaling challenging.
- Local Semantic Continuity: While SafeClean preserves gold spans, changes in surrounding non-gold regions might still disrupt local semantic continuity.
- BRIGHT-Specific Supervision: The pipeline's dependence on BRIGHT's span-level annotations limits its direct applicability to benchmarks without similar metadata.
- Future Work: The authors suggest that the MARCUS pipeline and BRIGHT+ corpus can be used to support future research on scalable, semantics-aware data preparation for reasoning-centric applications. They also recommend MARCUS-S as a default preprocessing strategy for production RAG systems due to its efficiency.
Summary updated