Transformers at SemEval-2024 Task 5: Legal Argument Reasoning Task in Civil Procedure using RoBERTa

Kriti Singhal, Jatin BediNAACL 2024

TL;DR

This paper tackles the novel task of Legal Argument Reasoning in Civil Procedure, aiming to automatically determine if a proposed legal solution is correct. The authors leverage a RoBERTa-based transformer model, fine-tuned on a dataset of U.S. civil procedure cases, to classify answers as right or wrong. Their submission achieved a respectable 13th rank in the SemEval-2024 competition, highlighting the potential of transformer models in specialized legal NLP tasks.

What Problem Does This Paper Address?

Mastering legal arguments requires deep domain knowledge and sophisticated reasoning skills, including analogical thinking. While NLP has made strides in understanding legal language, automatically assessing the correctness of a legal argument remains a significant challenge. This is particularly true in the legal domain, where publicly available datasets are scarce, making it difficult to train and evaluate robust models. The SemEval-2024 Task 5, "Legal Argument Reasoning Task in Civil Procedure," directly addresses this gap by introducing a new NLP task and a corresponding dataset focused on U.S. civil procedure. The goal is to classify whether a given solution to a legal question is correct or not.

Methods

The core of the authors' approach is the use of a transformer-based model, specifically RoBERTa (Robustly Optimized BERT Pretraining Approach). RoBERTa is a powerful pre-trained language model known for its strong performance on various NLP tasks. For this task, the authors employed the RoBERTa Large variant.

The methodology involved several key steps:

  • Data Imbalance Handling: The dataset exhibited an imbalance between the two classes (correct/incorrect answers). To mitigate this, the authors used minority sampling, randomly selecting instances from the majority class to match the number of instances in the minority class. This aimed to prevent bias in the trained model.
  • Fine-tuning: The pre-trained RoBERTa model was then fine-tuned on the legal argument reasoning dataset. Crucially, all available fields in the training data—the case introduction, the question, the proposed answer, and its detailed analysis—were used as input to predict the label (correct or incorrect).
  • Optimizer and Loss Function: The weighted Adam optimizer was used for training, along with cross-entropy loss as the objective function.
  • Training Parameters: The model was trained for 100 epochs with a batch size of 8 and a learning rate of 1e-5.

The paper also briefly mentions the concept of domain-specific adaptation of transformer models, citing examples like SciBERT and BioBERT, which were pre-trained on scientific and biomedical texts, respectively. While their primary submission used a general RoBERTa model, this context suggests an awareness of the benefits of specialized pre-training for legal domains.

Datasets

The primary dataset used in this paper was provided by the organizers of the SemEval-2024 Task 5: Legal Argument Reasoning Task in Civil Procedure.

  • Training Set: Contains 666 instances.
  • Class 0 (incorrect answers): 505 instances.
  • Class 1 (correct answers): 161 instances.
  • Each instance includes: a general introduction to a case, a question from that case, a possible argument solution, and a detailed analysis of why the argument is valid.
  • Test Set: Contains instances with a question, an answer, and an explanation, requiring the model to predict a label indicating whether the answer is right or wrong.

The dataset is based on the domain of U.S. civil procedure and is derived from a book aimed at law students.

Key Insights and Ideas

  • Transformer Models for Legal Reasoning: The paper demonstrates that transformer models like RoBERTa can be effectively applied to the novel and complex task of legal argument reasoning, even without extensive domain-specific pre-training.
  • Importance of Comprehensive Input: Utilizing all available textual information (case introduction, question, answer, and analysis) for fine-tuning proved crucial for the model's performance.
  • Data Imbalance is a Real Concern: The explicit mention of minority sampling highlights the common challenge of imbalanced datasets in specialized NLP tasks and the need for robust handling strategies.
  • Potential for Domain-Specific Models: While their submission used a general RoBERTa, the authors' discussion of models like Legal-BERT suggests a strong belief in the future benefits of transformers specifically pre-trained on legal corpora.

Key Results

The authors reported the following performance metrics for their RoBERTa-based submission:

  • RoBERTa Base: F1-Score: 0.5511, Accuracy: 0.6020
  • RoBERTa Large: F1-Score: 0.6172, Accuracy: 0.6531

The RoBERTa Large model, which was the primary model used for the submission, achieved an F1-score of 0.6172 and an accuracy of 0.6531.

Overall, the "Transformers" team secured the 13th rank in the Legal Argument Reasoning Task in Civil Procedure shared task, out of 21 participating teams.

Limitations and Future Work

  • RoBERTa Base vs. Large: Interestingly, the paper notes that the RoBERTa Base model, when fine-tuned with the same methodology, performed better than the RoBERTa Large model in terms of F1-score (0.6172 vs. 0.5511) and accuracy (0.6531 vs. 0.6020). This is a surprising result that warrants further investigation.
  • Domain-Specific Pre-training: The authors suggest that using transformers specifically trained on legal texts, such as Legal-BERT, could further improve performance.
  • Ensembling Techniques: They also propose exploring ensembling techniques by combining different transformer models, which has shown promise in other NLP tasks and could potentially boost results in legal reasoning.

Summary updated

BibTeX
@inproceedings{singhal2024transformers,
  author = {Kriti Singhal and Jatin Bedi},
  title = {{Transformers at SemEval-2024 Task 5: Legal Argument Reasoning Task in Civil Procedure using RoBERTa}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure