DUTh at SemEval 2024 Task 5: A multi-task learning approach for the Legal Argument Reasoning Task in Civil Procedure

Ioannis Maslaris, Avi ArampatzisNAACL 2024

TL;DR

This paper explores how to improve AI's ability to reason in complex legal scenarios by fine-tuning language models. They introduce a multi-task learning approach that trains models not only to predict the correct legal answer but also to generate the reasoning behind it, using human-written explanations. While their current models didn't achieve top performance, they highlight the potential of this method for future legal AI applications.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown impressive capabilities in understanding and generating human-like text. However, their reasoning abilities, especially in complex domains like law, are still a significant challenge. The SemEval 2024 Task 5 focuses on the Legal Argument Reasoning Task in Civil Procedure, aiming to build AI systems that can accurately answer multiple-choice questions about US civil procedure. This task is particularly difficult because legal texts are dense with detail, and understanding the nuances of legal arguments requires sophisticated reasoning. The dataset provides human-generated rationales for answers, offering a valuable resource for training AI, but effectively leveraging these rationales for improved reasoning is an open problem.

Methods

The DUTh team experimented with two main approaches:

  1. LegalBERT as a Classifier: They fine-tuned a LegalBERT model, a language model specifically pre-trained on legal text, to act as a classifier. This model was trained to predict whether a given answer to a legal question was correct or incorrect. They also explored using the human-generated rationales as part of the input to the model.
  2. Multi-task Learning with Flan-T5: They implemented a multi-task learning framework using a smaller version of the Flan-T5 model. This approach trains the model to perform two tasks simultaneously:
  3. Label Prediction: Predict the correct answer (correct/wrong).
  4. Rationale Generation: Generate a coherent rationale explaining why the answer is correct or wrong. This is achieved using a custom loss function that balances the error from both tasks. They also experimented with "Chain-of-Thought" (CoT) prompting, using specific phrases to guide the model's reasoning process.

To handle the input length limitations of LLMs, they employed a sliding window mechanism, breaking down the input text into manageable chunks.

Datasets

The primary dataset used is from the SemEval 2024 Task 5: The Legal Argument Reasoning Task in Civil Procedure. This dataset is derived from Joseph Glannon's "The Glannon Guide To Civil Procedure" and includes:

  • Introduction: Context for the legal case.
  • Question: The legal question to be answered.
  • Possible Answers: Multiple-choice options.
  • Analysis: Human-written reasoning for each answer.
  • Explanation: A paragraph discussing the reasoning for all possible answers.

The training set contains 666 entries, the development set has 84, and the test set has 98. Notably, the test set excludes the "analysis" and "explanation" features, making it a more challenging evaluation.

Key Insights and Ideas

  • Rationales are Key, but Tricky: The paper strongly suggests that human-generated rationales are crucial for improving AI's legal reasoning. However, simply including them as input might not be enough; the model needs to learn how to use them effectively.
  • Multi-task Learning Shows Promise: Training a model to both predict an answer and generate its justification (multi-task learning) is a promising direction. It forces the model to internalize the reasoning process, potentially leading to better understanding.
  • LegalBERT's Specialization: LegalBERT, being pre-trained on legal text, showed competitive performance, suggesting that domain-specific pre-training is beneficial for legal AI tasks.
  • Model Size Matters: The authors acknowledge that their smaller Flan-T5 model might have struggled to grasp complex legal concepts due to its limited parameter count, highlighting the ongoing importance of model scale in complex reasoning.
  • CoT Prefixes Help: While their initial CoT attempts weren't groundbreaking, the task-specific prefixes used in the multi-task setup ("Predict:" and "Explain:") proved effective in guiding the model.

Key Results

  • LegalBERT Classifier: Achieved the highest F1 score of 0.5382 and an accuracy of 0.6837 among the single-task models.
  • Multi-task Flan-T5: Achieved an F1 score of 0.5324 and an accuracy of 0.6224. While its F1 score was close to LegalBERT, its accuracy was lower.
  • Other Models: BERT and DistilBERT showed higher accuracy scores (0.7245) but lower F1 scores compared to LegalBERT.
  • Qualitative Observation: The generated rationales from the multi-task Flan-T5 model "barely made any sense," indicating that while the model was trained to generate them, the quality was poor, likely due to model size limitations.

Limitations and Future Work

  • Hardware Constraints: The authors mention that hardware limitations led them to experiment with a smaller Flan-T5 model. They believe a larger model could better grasp the context of rationales and draw more effective associations.
  • Suboptimal Performance: The paper candidly states that their systems did not achieve significantly high performance, acknowledging the difficulties of the task.
  • Future Directions:
    • Experimenting with larger language models for the multi-task approach.
  • Exploring more efficient loss functions or weighting strategies for multi-task learning.
  • Investigating alternative methods for incorporating rationales beyond simply adding them to the input.

Summary updated

BibTeX
@inproceedings{maslaris2024duth,
  author = {Ioannis Maslaris and Avi Arampatzis},
  title = {{DUTh at SemEval 2024 Task 5: A multi-task learning approach for the Legal Argument Reasoning Task in Civil Procedure}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure