ignore at SemEval-2024 Task 5: A Legal Classification Model with Summary Generation and Contrastive Learning

Binjie Sun, Xiaobing ZhouNAACL 2024

TL;DR

This paper tackles the challenging task of legal argument reasoning in civil procedure by developing a classification model. They achieved a respectable 14th place in the SemEval-2024 competition by combining data augmentation, summary generation with GPT-3.5, and supervised contrastive learning on top of a Legal-BERT baseline, significantly outperforming the official baseline.

What Problem Does This Paper Address?

The paper addresses SemEval-2024 Task 5, which focuses on the Legal Argument Reasoning Task in Civil Procedure. This task requires models to determine the correctness of a legal argument candidate answer given a question and a brief introduction to the legal topic. The dataset, derived from a law textbook, presents a complex NLP challenge due to its domain-specific nature and relatively small size (only 666 training entries). Existing legal NLP models often struggle with such nuanced reasoning tasks, highlighting the need for advanced techniques to effectively process and understand legal texts.

Methods

The authors built their system upon the Legal-BERT foundation model, which is pre-trained on a diverse corpus of English legal texts. To enhance its performance, they implemented three key strategies:

  • Data Augmentation: They expanded the training data by combining existing fields (explanation, question, complete analysis) to create new positive samples and by pairing analyses of incorrect answers with answers from other questions to generate negative samples. This helped balance the dataset and enrich it.
  • Summary Generation: Recognizing that direct truncation of long legal texts could lose crucial information, they employed GPT-3.5 to generate concise summaries of the explanation and question fields. This approach proved more effective than direct or sliding window truncation for capturing key information.
  • Supervised Contrastive Learning: To better distinguish between similar and dissimilar legal arguments, they incorporated supervised contrastive learning. This method pulls samples belonging to the same class closer together in the embedding space while pushing apart samples from different classes, using anchor, positive, and negative samples derived from the dataset.

The final loss function combined cross-entropy loss with this supervised contrastive loss.

Datasets

The primary dataset used is from SemEval-2024 Task 5: The Legal Argument Reasoning Task in Civil Procedure. This dataset consists of legal texts from a law textbook, with a training set of 666 entries and a validation set of 84 entries. The test set contains 98 examples. The authors note that the dataset is relatively small, which motivated their use of data augmentation.

Key Insights and Ideas

  • GPT-3.5 for Legal Text Summarization is Powerful: The paper demonstrates that leveraging large language models like GPT-3.5 for generating summaries of legal explanations significantly improves performance compared to traditional truncation methods. This suggests that abstractive summarization can better capture the essence of complex legal reasoning.
  • Contrastive Learning Enhances Robustness: Supervised contrastive learning proved crucial for improving the model's ability to differentiate between correct and incorrect legal arguments. By learning to group similar arguments and separate dissimilar ones, the model becomes more robust to subtle variations.
  • Data Augmentation is Essential for Small Datasets: The significant performance boost from data augmentation underscores its importance when dealing with limited training data, especially in specialized domains like legal NLP.
  • Legal-BERT as a Strong Foundation: The choice of Legal-BERT as a baseline proved effective, providing a solid starting point for further optimizations.

Key Results

The authors' system achieved an F1 score of 0.551 on the test set, ranking 14th in the SemEval-2024 Task 5 competition. This represents an improvement of 0.1241 over the official baseline model. The individual strategies contributed to this improvement: data augmentation alone boosted performance, and the combination of data augmentation, summary generation, and supervised contrastive learning yielded the best results.

Limitations and Future Work

The authors acknowledge the relatively small size of the dataset as a challenge. For future work, they propose:

  • Incorporating Prior Knowledge: Guiding the model with domain-specific prior knowledge to improve its reasoning capabilities.
  • Domain-Specific Knowledge Integration: Further integrating knowledge from law school textbooks into the exercises and analyses.
  • Handling Long Texts: Exploring external tools to assist in processing and optimizing models for very long legal texts, as their current model has a token limit.

Summary updated

BibTeX
@inproceedings{sun2024ignore,
  author = {Binjie Sun and Xiaobing Zhou},
  title = {{ignore at SemEval-2024 Task 5: A Legal Classification Model with Summary Generation and Contrastive Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure