SemEval-2024 Task 5: Argument Reasoning in Civil Procedure

Lena Held, Ivan HabernalNAACL 2024

TL;DR

This paper introduces a new benchmark for evaluating the legal reasoning capabilities of AI models, specifically in U.S. civil procedure. The task challenges models to determine if a given answer to a legal question is correct, with the top-performing systems achieving an F1-score of 82.31% using large language models like GPT-4.

What Problem Does This Paper Address?

Aspiring lawyers need strong reasoning skills to navigate legal cases, which involves understanding legal principles, applying them to specific facts, and identifying logical inconsistencies. While AI has made strides in natural language understanding, evaluating its ability to perform complex legal reasoning remains a significant challenge. Existing legal question-answering datasets often focus on finding the correct answer but overlook the reasoning process. This paper addresses this gap by creating a task and dataset specifically designed to test and benchmark AI's legal reasoning capabilities in the domain of U.S. civil procedure.

Methods

The core of this work is SemEval-2024 Task 5, which frames legal reasoning as a binary classification problem: given a question, an introduction to the relevant legal topic, and a proposed answer, the model must predict whether the answer is correct or incorrect. The dataset is derived from "The Glannon Guide To Civil Procedure," a textbook designed for law students, providing high-quality, expert-written question-answer pairs with detailed explanations. The task is intentionally simplified to a yes/no prediction to avoid the complexities of multiple-choice formats and to directly probe reasoning abilities.

Datasets

The dataset for this task was created by parsing "The Glannon Guide To Civil Procedure" (Glannon, 2018). It consists of question, answer, and explanation triplets. The dataset was split into training (666 instances), development (84 instances), and testing (98 instances) partitions. A notable detail is that the test partition was curated to contain questions that appear later in the book, assuming they are generally harder. The authors also identified and analyzed potential data leakage from the development to the test set, which affected a small portion of the test data.

Key Insights and Ideas

  • LLMs Dominate Legal Reasoning: The top-performing systems overwhelmingly leveraged large language models (LLMs), particularly GPT-4, demonstrating their current superiority in this complex reasoning task.
  • Prompt Engineering is Key: Techniques like custom prompt designs, Chain-of-Thought prompting, and ensembling were crucial for achieving high performance, highlighting the importance of how models are instructed.
  • Domain-Specific Models Lag Behind: While models like Legal-BERT were explored, they were consistently outperformed by general-purpose LLMs, suggesting that broad reasoning capabilities are more advantageous than specialized legal models for this task.
  • Task Simplification Matters: Reframing the task as a binary classification (correct/incorrect) rather than a full multiple-choice question answering task proved effective, though some teams also found success by remodeling it back into a multiple-choice format.
  • Data Imbalance is a Challenge: The dataset is imbalanced, with significantly more incorrect answers than correct ones. This necessitates careful evaluation metrics like macro F1-score and poses a challenge for models, as correctly identifying correct answers was considerably harder.

Key Results

Twenty participants submitted solutions, with the best system achieving an F1-score of 82.31%. The top-ranked system, HW-TSC, utilized GPT-4 with custom prompt designs and a self-evaluation mechanism. Other high-performing systems also relied on GPT-4, often employing ensembling or Chain-of-Thought prompting. The average accuracy across all systems was not explicitly stated as a single number, but the leaderboard shows a range of accuracies and F1-scores. The majority baseline, which predicted all answers as incorrect, achieved an F1-score of 42.69%.

Limitations and Future Work

The authors acknowledge a potential limitation regarding data leakage, as the book, while not publicly available, might have been seen by LLMs during their pre-training. They also note that some legal answers might be outdated due to the book's edition, though they consider them correct as they are expert-validated.

Future work could involve:

  • Utilizing the "analysis" field in the dataset, which was not used in this competition, to better understand the reasoning steps.
  • Creating a modified dataset split to fully address the data leakage issue.
  • Exploring more sophisticated techniques to evaluate the reasoning process itself, rather than just the correctness of the answer.
  • Investigating how to improve the performance on correctly identifying correct answers, which proved significantly more challenging than identifying incorrect ones.

Summary updated

BibTeX
@inproceedings{held2024semeval,
  author = {Lena Held and Ivan Habernal},
  title = {{SemEval-2024 Task 5: Argument Reasoning in Civil Procedure}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure