Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4

Dan Schumacher, Anthony RiosNAACL 2024

TL;DR

This paper presents a novel prompt-based approach using GPT-4 for legal argument reasoning in civil procedures, achieving 5th place in the SemEval 2024 Task 5. By ensembling various prompting strategies, including retrieval-augmented generation and chain-of-thought, the team significantly improved performance over baseline methods.

What Problem Does This Paper Address?

Legal argument reasoning is a crucial skill for law students, requiring them to analyze complex legal texts and determine the validity of arguments. Developing AI systems that can perform this task is challenging due to the need for natural language understanding and logical inference within domain-specific contexts. Existing NLP approaches for legal reasoning fall into two main categories: fine-tuning pre-trained models (like LegalBERT) and leveraging large language models (LLMs). While LLMs show promise, optimizing their performance for specific tasks like legal reasoning often requires careful prompt engineering. This paper addresses the challenge of improving LLM performance on legal argument reasoning by exploring and ensembling various prompting strategies.

Methods

The core of the UTSA-NLP system is a prompt-based approach using GPT-4. They explore several prompting strategies:

  • Zero-shot Prompting: The model receives only the system prompt and the test instance without any examples.
  • Few-shot Prompting: The model is provided with a small number of examples to guide its reasoning.
  • Chain-of-Thought (CoT) Prompting: This involves prompting the model to generate an intermediate "Analysis" section that explains its reasoning before providing a final label.
  • Retrieval-Augmented Generation (RAG): For few-shot examples, instead of using arbitrary examples, they employ a retrieval mechanism (using LegalBERT embeddings and cosine similarity) to find the most relevant examples from the training data for each test instance.

The paper also emphasizes ensembling these strategies. They combine predictions from multiple models (Zero-shot, Zero-shot & CoT, Few-shot, Few-shot & CoT, Few-shot & RAG) using a voting mechanism with a threshold to arrive at the final prediction.

Datasets

The primary dataset used in this work is from SemEval Task 5, The Legal Argument Reasoning Task in Civil Procedure Challenge. This dataset was scraped from "The Glannon Guide To Civil Procedure," a textbook for law students. Each instance in the dataset consists of an Introduction (general background), a Question, an Answer Candidate, an Analysis (expert-defined explanation), and a Label (TRUE/FALSE).

Key Insights and Ideas

  • Prompt Ensembling is Key: The most significant finding is that ensembling various prompting strategies, particularly combining retrieval-augmented few-shot learning with chain-of-thought, leads to substantial performance gains.
  • RAG and CoT are Crucial: Removing the Retrieval-Augmented Generation (RAG) component resulted in the largest performance drop, indicating the importance of relevant examples. Similarly, removing Chain-of-Thought (CoT) also significantly impacted performance.
  • LLMs Struggle with Flawed Reasoning: GPT-4 sometimes incorrectly predicts an answer as TRUE when the answer candidate's reasoning is flawed, even if the core conclusion is correct. This highlights a nuanced challenge in evaluating not just the correctness of an answer but also the validity of its justification.
  • Overfitting Concerns: The authors note that their validation results might be overestimated due to overlap with the training data and potential overfitting from optimizing the ensemble on the validation set.

Key Results

The team achieved a Macro F1 score of .8095 on the validation dataset and .7315 (5th out of 21 teams) on the final test set.

  • Validation Performance: The proposed "Few-Shot & CoT & RAG" method achieved a Macro F1 of .7306, outperforming other GPT-4 based methods and baselines like Legal-BERT and RoBERTa. The final ensemble further boosted this to .8095.
  • Competition Performance: The ensemble achieved a Macro F1 of .7315 in the competition.
  • Error Analysis: The majority of errors were false positives (10 out of 12 on the validation set). Common error types included incorrect reasoning, similar language between question/introduction and answer, and instances where the model was misled by extraneous information in long introductions.

Limitations and Future Work

  • Analysis Section Format: The current "Analysis" section for CoT reasoning doesn't strictly follow step-by-step explanations. Future work could involve rewording this section to be more explicit.
  • Number of In-Context Examples: The study was limited to 2 in-context examples due to API costs. Increasing this number could potentially improve performance.
  • Closed-Source Model Dependency: The reliance on GPT-4 is a limitation. Future research should explore open-source models (e.g., T5, Llama 2) to assess the impact of pretraining data and model architecture on legal reasoning performance.
  • Validation Set Overlap: The authors acknowledge that the validation set might have had significant overlap with the training set, potentially leading to an overestimation of performance. They suggest generating a better validation split for future work.

Summary updated

BibTeX
@inproceedings{schumacher2024team,
  author = {Dan Schumacher and Anthony Rios},
  title = {{Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure