TL;DR
This paper introduces Ko-LongRAG, the first benchmark for evaluating Korean long-context Retrieval-Augmented Generation (RAG) systems. It pioneers a "retrieval-free" approach to construct high-quality RAG datasets, overcoming the limitations of existing English-centric benchmarks and the challenges of low-resource languages.
What Problem Does This Paper Address?
The rapid advancements in Large Language Models (LLMs) have significantly boosted the capabilities of RAG systems, enabling them to process and comprehend extended texts for tasks like document summarization and question answering. However, the evaluation landscape for RAG is heavily skewed towards English, leaving a critical gap for low-resource languages like Korean. Existing benchmarks often rely on external retrieval systems, which are difficult to implement and maintain for languages with less developed knowledge bases and research infrastructure. Furthermore, Korean's linguistic complexities, such as rich morphology and flexible word order, pose unique challenges for traditional retrieval methods. This paper aims to address this by creating a comprehensive and accessible RAG benchmark specifically for Korean long-context scenarios.
Methods
The core innovation of Ko-LongRAG lies in its "retrieval-free" approach to dataset construction. Instead of relying on external retrieval systems, the authors developed a method centered around Specialized Content Knowledge (SCK). This involves:
- Document Clustering: The Korean Wiki Raw Corpus is processed to extract SCK (domain, central concepts, key words) using an LLM. Documents are then clustered into domains based on this SCK, ensuring thematic coherence.
- Long-context QA Generation: Within these clusters, LLMs (specifically GPT-4o) are used to generate question-answer pairs. This process simulates retrieval-based scenarios by ensuring questions are answerable from the clustered documents, maintaining high relevance and contextual fidelity. Two types of QA are generated:
- Single-document QA: Questions requiring precise extraction from a single document.
- Multi-document QA: Questions requiring reasoning across multiple documents, categorized into "comparison" (contrasting information) and "bridge" (logical inference) types.
- Manual Quality Assurance: A rigorous manual review process, guided by a detailed checklist and involving multiple annotators, ensures the quality and reliability of the generated QA pairs. This includes verifying question clarity, answer accuracy, and context integrity.
- Incorporating Unanswerable Cases: The benchmark intentionally includes "unanswerable" cases by pairing irrelevant questions with documents, testing the models' ability to recognize when an answer cannot be derived from the provided context.
Datasets
The primary dataset introduced is Ko-LongRAG, a novel benchmark for Korean long-context RAG. It is constructed using the Korean Wiki Raw Corpus. The benchmark consists of:
- 300 single-document QA questions with an average context length of 2,915 tokens.
- 300 multi-document QA questions with an average context length of 14,092 tokens.
- Approximately 16.6% of the dataset comprises unanswerable questions.
The dataset is made publicly available.
Key Insights and Ideas
- Retrieval-Free RAG Evaluation is Feasible and Beneficial: The success of Ko-LongRAG demonstrates that high-quality RAG benchmarks can be built without complex retrieval infrastructure, making them more accessible for low-resource languages.
- SCK is Key for Controlled QA Generation: Leveraging Specialized Content Knowledge (SCK) allows for systematic document clustering and targeted QA generation, ensuring contextual relevance and diversity.
- Korean Language Nuances Require Tailored Benchmarks: The paper highlights that existing English-centric benchmarks are insufficient for evaluating Korean RAG due to linguistic specificities. Ko-LongRAG provides a much-needed specialized evaluation tool.
- Unanswerable Cases are Crucial for Robustness: The inclusion of unanswerable questions is vital for assessing a model's true understanding and its ability to avoid hallucination when information is absent.
Key Results
The paper evaluates various proprietary and open-sourced LLMs on Ko-LongRAG.
- Proprietary Models: OpenAI's
o1model achieved the highest performance. - Open-Sourced Models: EXAONE 3.5 (various parameter sizes) demonstrated superior performance among open-sourced models, consistently outperforming others, especially in Korean-specialized categories.
- Model Size Matters: A strong positive correlation was observed between model size (parameter count) and performance, particularly for open-sourced models.
- Multi-document QA is Harder: Multi-document QA tasks generally resulted in lower scores than single-document QA, indicating the increased difficulty of complex reasoning.
- Answerability Performance Varies: Proprietary models generally performed well on unanswerable cases, while many open-sourced models struggled, with EXAONE 3.5 and Qwen 2.5 being notable exceptions.
Limitations and Future Work
- Language Scope: While Ko-LongRAG is tailored for Korean, its broader applicability to other typologically distinct languages needs further exploration. Extending the methodology to multilingual RAG systems is a key future direction.
- Potential Distributional Bias: The use of GPT-4o for initial QA generation could introduce a slight bias. However, the authors mitigated this by conducting comparative experiments with other proprietary models (Claude-3.5 Sonnet, Gemini 2.5 Flash), finding that overall ranking trends remained consistent, especially for open-source models.
- Future Work: The authors suggest that Ko-LongRAG can drive innovation in multilingual RAG research and advocate for the development of more equitable, language-inclusive benchmarks. Further research could explore more sophisticated methods for generating diverse and challenging QA pairs.
Summary updated