ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models

Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah, Reem AlZahrani, Hebah AlShamlan, Omar Knio, George TurkiyyahACL 2024

TL;DR

This paper introduces ArabLegalEval, a novel benchmark designed to evaluate the legal reasoning capabilities of Large Language Models (LLMs) in Arabic. By creating a diverse set of tasks derived from Saudi legal documents and translated benchmarks, they aim to bridge the gap in evaluating LLMs for non-English legal domains and pave the way for more robust Arabic legal AI.

What Problem Does This Paper Address?

The rapid advancement of LLMs has shown impressive capabilities across various domains, but their performance in specialized fields, particularly in low-resource languages like Arabic, remains underexplored. Specifically, evaluating how well LLMs understand and process Arabic legal texts is crucial for their adoption in legal contexts. Existing benchmarks are often limited to English or general reasoning, leaving a significant gap in assessing Arabic legal knowledge. This paper aims to address this by developing a comprehensive benchmark to evaluate LLMs' proficiency in Arabic legal reasoning and knowledge recall.

Methods

The authors developed ArabLegalEval through a multi-pronged approach:

  • Data Curation: They sourced raw Arabic legal data from the Saudi Ministry of Justice (MoJ) and the Board of Experts (BoE), comprising regulations, statutes, and circulars. They also incorporated human-written FAQs and translated subsets of existing English legal benchmarks like LegalBench.
  • Task Generation:
    • MCQs: They employed LLMs to generate Multiple Choice Questions (MCQs) from legal documents. This involved methods like "QA to MCQ" and "Chain of Thought (CoT)" prompting, and in-context learning with examples from ArabicMMLU to ensure style and format consistency.
  • QA: They created an open-ended Question Answering (QA) task using filtered and verified human-written FAQs, focusing on questions referencing specific legal statutes and articles.
  • LegalBench Translation: They translated key datasets from the English LegalBench benchmark into Arabic, ensuring minimal localization requirements for universality.
  • Evaluation Framework: They used LLMs as judges to evaluate answer similarity on a scale of 0-5, focusing on semantic understanding rather than just exact matches. They also employed standard metrics like ROUGE for translation quality assessment.

Datasets

  • ArabLegalEval (New Benchmark): This is the primary contribution, a multitask benchmark for Arabic legal knowledge. It comprises:
  • MCQs: Over 10,000 questions generated from MoJ and BoE documents.
  • NajizQA: A set of QA pairs derived from human-written FAQs.
  • Translated LegalBench: Arabic translations of selected LegalBench datasets (Consumer Contracts QA, Contracts QA, Privacy Policy QA, Privacy Policy Entailment).
  • Source Documents:
    • Saudi Ministry of Justice (MoJ) documents (regulations, circulars).
  • Board of Experts (BoE) documents (rules).
  • Human-written FAQs.
  • English LegalBench datasets (translated).
  • ArabicMMLU (for inspiration and few-shot examples).
  • United Nations Parallel Corpus (for translation model evaluation).

Key Insights and Ideas

  • LLMs as Legal Assistants: The paper highlights the growing potential of LLMs to act as legal assistants, aiding in research, drafting, and summarization.
  • Multilingual Reasoning Emergence: The performance of LLMs like GPT-4 on Arabic tasks suggests that multilingual reasoning capabilities emerge with increased model scale, even without explicit Arabic-centric training.
  • Prompting Matters: Few-shot prompting, especially with Chain-of-Thought (CoT) reasoning, significantly boosts LLM performance on legal tasks, indicating the importance of carefully crafted prompts.
  • LLM-as-a-Judge: Using LLMs (like GPT-4) as judges for evaluating QA tasks is a viable approach, though care must be taken to account for language differences and potential biases.
  • Translation Quality: While several machine translation models perform reasonably well, Opus MT demonstrated superior quality for legal texts, as validated by both GPT-4 and human legal experts.

Key Results

  • ArabLegalEval Performance: The benchmark was used to evaluate several LLMs, including GPT-4, Command R/R+, and Llama3. GPT-40 showed strong performance, achieving up to 79.10% accuracy on the generated MCQs with few-shot prompting.
  • LegalBench Evaluation: Command R Plus achieved the highest F1 scores on several LegalBench tasks, reaching 99% on Contract QA with few-shot learning and 66% on Privacy Policy Entailment with one-shot learning. GPT-4 also demonstrated strong performance, particularly on Privacy Policy QA.
  • Human Baseline: Legal experts achieved an F1 score of 0.9 for MCQs, 0.65 for NajizQA, and 0.73 for Arabic LegalBench, providing a benchmark for LLM performance.
  • Translation: Opus MT achieved the highest ROUGE scores (0.519 for ROUGE-1, 0.308 for ROUGE-2, and 0.509 for ROUGE-L) in translating the UN Parallel Corpus, and was favored by human experts for its translation quality.

Limitations and Future Work

  • Geographic Representation: The current benchmark heavily relies on Saudi Arabian legal documents. Future work should include documents from other Arabic-speaking countries to improve geographic representation.
  • Model Coverage: The study did not evaluate all available LLMs, limiting generalizability. Future work should include a broader range of models.
  • Expert Validation: Limited access to legal experts affected validation depth. Involving more experts would enhance quality control.
  • Dataset Granularity: The dataset lacks granular categorization (e.g., task-specific prior knowledge, document origin, AI-generated content labels). Adding such metadata would aid nuanced model training and evaluation.
  • Future Plans: The authors plan to expand the benchmark by incorporating more KSA regulatory documents, court cases, and judicial decisions to further enhance it and promote advancements in Arabic legal AI.

Summary updated

BibTeX
@inproceedings{hijazi2024arablegaleval,
  author = {Faris Hijazi and Somayah AlHarbi and Abdulaziz AlHussein and Harethah Abu Shairah and Reem AlZahrani and Hebah AlShamlan and Omar Knio and George Turkiyyah},
  title = {{ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure