Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models

Kimyeeun Kimyeeun, Choi Youngrok, Eunkyung Choi, Jinhwan Choi, Hai Jin Park, Wonseok HwangEMNLP 2024 · DOI

Introduction

While Large Language Models (LLMs) have demonstrated remarkable proficiency in passing the U.S. Bar Exam, legal expertise is not a monolithic capability. Legal systems are deeply rooted in specific jurisdictions, languages, and cultural contexts. Most existing legal benchmarks focus on the English language and the U.S. Common Law system. In contrast, South Korea operates under a Civil Law system with its own unique complexities.

Existing Korean benchmarks, such as KMMLU, often include legal questions sourced from general license exams, which lack the professional depth required for actual legal practice. Furthermore, these benchmarks typically evaluate models in "closed-book" settings. This does not reflect the reality of legal work, where practitioners rely heavily on consulting statutes and precedents. To address this gap, we introduce KBL (Korean Benchmark for Legal Language Understanding), a rigorous, expert-validated benchmark designed to test both internal knowledge and the ability to apply retrieved legal information.

Methods and Evaluation Framework

The KBL benchmark was developed in collaboration with eight licensed lawyers to ensure professional relevance and accuracy. The evaluation framework utilizes two primary settings:

  1. Closed-Book Setting: Models answer questions based solely on their pre-trained parameters in a zero-shot configuration.
  2. Open-Book (RAG) Setting: Models are integrated with a BM25-based retriever, granting them access to a comprehensive corpus of 11,000 active Korean statutes and 150,000 precedents.

To ensure fair comparison among open-source models, the researchers employed logit-based evaluation. By comparing the probabilities assigned to multiple-choice options rather than relying on direct text generation, the evaluation remains stable and less susceptible to the noise often found in generated outputs.

Datasets

The KBL suite comprises several high-quality datasets categorized into Knowledge, Reasoning, and professional examination tasks. To maintain the integrity of the benchmark, the authors performed "fuzzy matching" against existing datasets like KMMLU to eliminate any potential overlap.

Table 1: Detailed statistics of the KBL benchmark datasets, including sample counts and token lengths for various legal tasks.

As shown in Table 1, the benchmark includes:

  • Legal Knowledge Tasks (510 examples): Covering Legal Concept QA, Offense Component QA (matching actions to specific crimes), and Statute Matching.
  • Legal Reasoning Tasks (288 examples): Including Causal Reasoning (causality between actions and results), Statement Consistency, and Case Relevance.
  • Korean Bar Exam (2,510 examples): A comprehensive set of multiple-choice questions from 2012–2024 spanning Public, Civil, and Criminal law.
  • RAG Corpus: A massive collection of 220,000 statute articles and 150,000 precedents.

Key Insights and Analysis

The "Application" Wall

A critical finding of this study is the distinction between "Rule-type" questions (recalling the law) and "Application-type" questions (applying law to specific facts). While Retrieval-Augmented Generation (RAG) significantly improves performance on Rule-type questions, it offers minimal benefit—and in some cases, even hinders—performance on Application-type questions. This suggests that the primary bottleneck for LLMs in legal contexts is logical reasoning rather than information retrieval.

Data Quality and Expert Verification

During the development of KBL, the researchers discovered that publicly available legal data created by non-experts contained error rates as high as 21%. This highlights the absolute necessity of professional legal verification when building benchmarks for high-stakes domains like law.

RAG is Not a Universal Solution

The interaction between a model's reasoning capabilities and retrieved context is highly sensitive. While RAG generally helped GPT-4, it led to performance degradation in certain Claude-3 models. This indicates that simply providing more context does not guarantee better legal decision-making.

Key Results

Table 2: Performance metrics of proprietary and open-source LLMs on the KBL benchmark in both closed-book and open-book (RAG) configurations.

As detailed in Table 2, GPT-4 remains the leading model for Korean legal tasks, achieving 48.1% accuracy on the 2024 Bar Exam in a closed-book setting. In the open-book configuration, GPT-4's performance improved by up to 8.6% on specific tasks, though results varied across different legal domains. Notably, open-source models like Qwen2-72B demonstrated significant progress, outperforming GPT-3.5 and narrowing the gap with top-tier proprietary models in legal knowledge tasks. However, no model has yet reached a "passing" level comparable to their performance on the U.S. Bar Exam, signaling that Korean legal language remains a formidable challenge.

Limitations and Future Work

KBL currently focuses on multiple-choice questions, which allows for objective scoring but does not evaluate a model's ability to draft legal documents or construct complex narrative arguments. Additionally, the RAG implementation used a basic BM25 retriever; future iterations may benefit from more sophisticated retrieval pipelines, such as hybrid search or re-ranking. Future research will focus on expanding KBL to include generative tasks and investigating the impact of specialized fine-tuning on Korean legal corpora.

Summary updated

BibTeX
@inproceedings{kimyeeun2024developing,
  author = {Kimyeeun Kimyeeun and Choi Youngrok and Eunkyung Choi and Jinhwan Choi and Hai Jin Park and Wonseok Hwang},
  title = {{Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.319},
}
Figure