SU-FMI at SemEval-2024 Task 5: From BERT Fine-Tuning to LLM Prompt Engineering - Approaches in Legal Argument Reasoning

Kristiyan Krumov, Svetla Boytcheva, Ivan KoytchevNAACL 2024

TL;DR

This paper tackles the challenging task of legal argument reasoning by exploring both fine-tuned BERT models and the novel application of Large Language Models (LLMs) through prompt engineering. Their hybrid approach achieved a 3rd place ranking in the SemEval-2024 Task 5, demonstrating the significant potential of LLMs in understanding and reasoning over complex legal texts.

What Problem Does This Paper Address?

Legal professionals deal with a massive volume of complex documents, making analysis and reasoning a significant challenge. Natural Language Understanding (NLU) technologies offer a potential solution, but adapting them to the specialized language and intricate reasoning required in the legal domain remains difficult. Specifically, the SemEval-2024 Task 5 focuses on legal argument reasoning, a task that requires understanding case introductions, legal questions, proposed solutions, and detailed analyses to determine the correctness of an argument. This task is particularly challenging due to the length of legal texts and the nuanced reasoning involved, pushing the boundaries of existing NLP models.

Methods

The authors explored two main avenues for tackling legal argument reasoning:

  • Fine-tuned Transformer Models: They started with pre-trained transformer models like BERT and its legal-specific variants (Legal-BERT, Legal-RoBERTa). To handle the long sequences common in legal documents, they experimented with models like Longformer, which have extended context windows. They also investigated summarization techniques (BART, LexRank, ChatGPT) to condense lengthy texts before feeding them into classification models. Additionally, they fine-tuned models on a custom-tailored U.S. Civil Procedure dataset to enhance their relational and contextual understanding of legal language.
  • LLM + Legal Prompt Engineering (LPE): This was a more innovative approach. Instead of traditional fine-tuning, they leveraged the generative capabilities of LLMs (Mistral-7b-Instruct, Llama2-70b, GPT-3.5-Turbo, GPT-4) by crafting specific prompts. The goal was to guide the LLMs to perform a binary classification (TRUE/FALSE) based on the provided legal context. They experimented with various prompt structures, including "chain-of-thought" prompting, to elicit more reasoned responses.

Datasets

The primary dataset used was the SemEval-2024 Task 5 dataset, which consists of instances containing a case introduction, a specific legal question, a proposed solution argument, and a detailed analysis. The dataset is used for a binary classification task to determine the correctness of an argument.

They also created a custom-tailored U.S. Civil Procedure dataset for fine-tuning transformer models. This dataset was constructed by extracting keywords from explanations and questions, generating search queries, and downloading relevant cases from the Caselaw Access Project, resulting in 1985 legal texts.

Key Insights and Ideas

  • LLMs Shine in Legal Reasoning: The most significant insight is the superior performance of LLMs, particularly GPT-3.5-Turbo and GPT-4, when combined with Legal Prompt Engineering (LPE). These models significantly outperformed traditional transformer models, highlighting their advanced reasoning capabilities.
  • Prompt Engineering is Key: The effectiveness of LLMs was heavily dependent on the quality and structure of the prompts. Crafting specific prompts that guide the LLM to focus on the task and provide the desired output format (TRUE/FALSE) was crucial.
  • Handling Long Texts is Essential: Legal documents are inherently long. While transformer models have limitations with sequence length, techniques like Longformer and summarization showed promise in mitigating these issues.
  • Legal-Specific Models Offer a Strong Baseline: Pre-trained models on legal corpora (e.g., Legal-BERT, Legal-RoBERTa) provided a solid foundation and competitive performance, especially when fine-tuned.
  • Summarization Aids Understanding: Condensing long legal texts into summaries helped reduce processing complexity and potentially improve the focus on relevant information for downstream tasks.

Key Results

The paper reports a 3rd place ranking on the SemEval-2024 Task 5 Leaderboard out of 20 participating teams.

  • LLM Performance: GPT-4 achieved the highest Macro-F1 score of 0.7728 on the test set, followed closely by GPT-3.5-Turbo with 0.60. Mistral-7b-instruct and Llama2-70b also showed strong performance with LPE, achieving 0.58 and 0.59 respectively.
  • Transformer Model Performance: Fine-tuned transformer models, especially those incorporating summarization (e.g., Legal-RoBERTa + BART, SU-FMI-LegalBERT + BART), achieved Macro-F1 scores in the range of 0.50-0.54. Models using Sliding Window techniques also performed reasonably well, with SU-FMI-LegalBERT + SW reaching 0.60.
  • Baselines: The baseline models (Majority and Random) achieved significantly lower Macro-F1 scores (0.42-0.46), underscoring the complexity of the task.

Limitations and Future Work

  • LLM Output Consistency: While GPT-3/4 consistently followed the TRUE/FALSE output instruction, Mistral-7b-instruct often included additional explanations, requiring more complex post-processing.
  • Hierarchical Transformer Exploration: The authors mention initial experiments with a hierarchical transformer variant (HIER-BERT) that did not yield very high results (0.47 f1-macro on the dev partition) due to setup complexity. They suggest this architecture could be a direction for future research.
  • Generalization of LPE: While LPE showed great promise, further investigation into its broader applicability and robustness across different legal tasks and LLMs is warranted.
  • Complexity of Legal Reasoning: The authors acknowledge that the inherent complexity of legal reasoning continues to pose ongoing challenges, even with advanced models.

Summary updated

BibTeX
@inproceedings{krumov2024su,
  author = {Kristiyan Krumov and Svetla Boytcheva and Ivan Koytchev},
  title = {{SU-FMI at SemEval-2024 Task 5: From BERT Fine-Tuning to LLM Prompt Engineering - Approaches in Legal Argument Reasoning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure