Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in passing the U.S. Bar Exam, legal expertise is not a monolithic capability. Legal systems are deeply rooted in specific jurisdictions, languages, and cultural contexts. Most existing legal benchmarks focus on the English language and the U.S. Common Law system. In contrast, South Korea operates under a Civil Law system with its own unique complexities.
Existing Korean benchmarks, such as KMMLU, often include legal questions sourced from general license exams, which lack the professional depth required for actual legal practice. Furthermore, these benchmarks typically evaluate models in "closed-book" settings. This does not reflect the reality of legal work, where practitioners rely heavily on consulting statutes and precedents. To address this gap, we introduce KBL (Korean Benchmark for Legal Language Understanding), a rigorous, expert-validated benchmark designed to test both internal knowledge and the ability to apply retrieved legal information.
Methods and Evaluation Framework
The KBL benchmark was developed in collaboration with eight licensed lawyers to ensure professional relevance and accuracy. The evaluation framework utilizes two primary settings:
- Closed-Book Setting: Models answer questions based solely on their pre-trained parameters in a zero-shot configuration.
- Open-Book (RAG) Setting: Models are integrated with a BM25-based retriever, granting them access to a comprehensive corpus of 11,000 active Korean statutes and 150,000 precedents.
To ensure fair comparison among open-source models, the researchers employed logit-based evaluation. By comparing the probabilities assigned to multiple-choice options rather than relying on direct text generation, the evaluation remains stable and less susceptible to the noise often found in generated outputs.
Datasets
The KBL suite comprises several high-quality datasets categorized into Knowledge, Reasoning, and professional examination tasks. To maintain the integrity of the benchmark, the authors performed "fuzzy matching" against existing datasets like KMMLU to eliminate any potential overlap.
As shown in Table 1, the benchmark includes:
- Legal Knowledge Tasks (510 examples): Covering Legal Concept QA, Offense Component QA (matching actions to specific crimes), and Statute Matching.
- Legal Reasoning Tasks (288 examples): Including Causal Reasoning (causality between actions and results), Statement Consistency, and Case Relevance.
- Korean Bar Exam (2,510 examples): A comprehensive set of multiple-choice questions from 2012–2024 spanning Public, Civil, and Criminal law.
- RAG Corpus: A massive collection of 220,000 statute articles and 150,000 precedents.
Key Insights and Analysis
The "Application" Wall
A critical finding of this study is the distinction between "Rule-type" questions (recalling the law) and "Application-type" questions (applying law to specific facts). While Retrieval-Augmented Generation (RAG) significantly improves performance on Rule-type questions, it offers minimal benefit—and in some cases, even hinders—performance on Application-type questions. This suggests that the primary bottleneck for LLMs in legal contexts is logical reasoning rather than information retrieval.
Data Quality and Expert Verification
During the development of KBL, the researchers discovered that publicly available legal data created by non-experts contained error rates as high as 21%. This highlights the absolute necessity of professional legal verification when building benchmarks for high-stakes domains like law.
RAG is Not a Universal Solution
The interaction between a model's reasoning capabilities and retrieved context is highly sensitive. While RAG generally helped GPT-4, it led to performance degradation in certain Claude-3 models. This indicates that simply providing more context does not guarantee better legal decision-making.
Key Results
As detailed in Table 2, GPT-4 remains the leading model for Korean legal tasks, achieving 48.1% accuracy on the 2024 Bar Exam in a closed-book setting. In the open-book configuration, GPT-4's performance improved by up to 8.6% on specific tasks, though results varied across different legal domains. Notably, open-source models like Qwen2-72B demonstrated significant progress, outperforming GPT-3.5 and narrowing the gap with top-tier proprietary models in legal knowledge tasks. However, no model has yet reached a "passing" level comparable to their performance on the U.S. Bar Exam, signaling that Korean legal language remains a formidable challenge.
Limitations and Future Work
KBL currently focuses on multiple-choice questions, which allows for objective scoring but does not evaluate a model's ability to draft legal documents or construct complex narrative arguments. Additionally, the RAG implementation used a basic BM25 retriever; future iterations may benefit from more sophisticated retrieval pipelines, such as hybrid search or re-ranking. Future research will focus on expanding KBL to include generative tasks and investigating the impact of specialized fine-tuning on Korean legal corpora.
Summary updated