TL;DR
This paper introduces EKRAG, a new benchmark dataset for evaluating Retrieval-Augmented Generation (RAG) systems in enterprise knowledge question answering. They also propose novel "embedding-model-as-judge" and "ranking-model-as-judge" evaluation methods to complement existing LLM-based approaches, offering a more nuanced understanding of RAG performance in corporate settings.
What Problem Does This Paper Address?
Large Language Models (LLMs) are powerful, but their static knowledge bases limit their ability to provide accurate, up-to-date answers to complex, domain-specific questions, especially in dynamic enterprise environments. Retrieval-Augmented Generation (RAG) aims to solve this by grounding LLMs with external knowledge. However, evaluating RAG systems, particularly in enterprise contexts where factuality and context awareness are paramount, is challenging. Existing RAG benchmarks often focus on general domains, leaving a gap in understanding how RAG performs with diverse corporate documents like financial reports and technical blogs. This paper addresses this gap by creating a specialized benchmark and evaluation methodologies for enterprise RAG.
Methods
The authors introduce the EKRAG dataset, a collection of 1,347 manually curated questions derived from a corpus of 5,000 corporate documents. These questions cover five core types (extractive QA, abstractive QA, summarization, financial numerical reasoning, and no-answer questions) and often require multi-hop reasoning across documents.
To evaluate RAG systems, they propose two novel judging mechanisms:
- Embedding-model-as-judge: This method uses the similarity of embeddings between generated answers and ground truth answers/contexts to assess correctness, relevance, and faithfulness.
- Ranking-model-as-judge: This approach leverages a cross-encoder model to score the relevance between generated answers and ground truth answers/contexts, providing a more nuanced evaluation than simple embedding similarity.
These methods are used alongside the established LLM-as-judge approach for a comprehensive evaluation. They also explore various retrieval strategies, including dense, sparse, and hybrid search, and investigate the impact of embedding models and reranking.
Datasets
- EKRAG Dataset: A newly introduced benchmark dataset consisting of 1,347 manually curated questions, relevant context chunks, referenced documents, and ground truth answers. It's derived from 5,000 corporate documents (technical blogs, news releases, SEC filings, leadership communications).
- Existing Benchmarks: The paper references previous RAG evaluation works that used existing QA datasets for general domains.
Key Insights and Ideas
- Enterprise RAG is different: Corporate environments demand higher factuality and timeliness, making RAG evaluation more critical and complex than in general domains.
- Novel evaluation metrics are needed: Relying solely on LLM-as-judge might not fully capture RAG performance. Embedding- and ranking-model-based judges offer complementary perspectives on answer quality.
- Standard RAG techniques don't always translate: Surprisingly, well-known techniques like HYDE and hybrid search did not significantly improve enterprise knowledge retrieval accuracy in their experiments.
- Multi-embedding vectors can be effective: Concatenating embeddings from different models (e.g., Arctic-L and NV-EMB) showed promise for improving retrieval.
- Reranking is crucial: A dedicated reranking model significantly improved recall across all tested embedding models, highlighting its importance in the RAG pipeline.
- Chunk count matters: The number of retrieved chunks impacts correctness, with an optimal window existing between 1 and 5 chunks; too many can introduce noise.
Key Results
- The NV-EMB embedding model significantly outperformed other tested models on public benchmarks and in their EKRAG evaluation.
- Reranking consistently and significantly improved the recall of all embedding models, with QA-Mistral-4B being used as the default reranker.
- The proposed embedding-model-as-judge and ranking-model-as-judge methods provided valuable insights into answer quality, complementing LLM-as-judge.
- HYDE and hybrid search did not consistently improve retrieval accuracy for enterprise knowledge.
- For answer generation, larger LLMs did not show a substantial performance advantage over smaller ones like Llama3-8B in this specific task, suggesting a balance between performance and efficiency.
- The absence of retrieved chunks significantly diminished answer correctness, validating the RAG approach.
Limitations and Future Work
- Dataset Scope: The current EKRAG dataset (v.1) utilizes only "Corporate News and Blogs" and "Corporate Technical Blogs" categories. Future work could expand to include other document types like SEC filings and leadership communications more broadly.
- Evaluation Nuances: While the proposed judge mechanisms offer new insights, further research could explore more sophisticated ways to combine their outputs or develop even more robust evaluation metrics.
- Retrieval Strategy Optimization: The paper found that some standard retrieval strategies didn't perform as expected. Future work could focus on developing or adapting retrieval methods specifically for the nuances of enterprise data.
- Computational Cost: The paper notes that some high-performing embedding models (like NV-EMB) come with high computational costs, suggesting a need for more efficient alternatives.
Summary updated