TL;DR
This paper tackles the challenging task of legal argument reasoning in civil procedure, where models must determine if a candidate answer is correct given a legal topic and question. The authors propose a system that fine-tunes Legal-BERT, a specialized BERT model for legal text, and enhances it with R-Drop for data augmentation and focal loss to handle imbalanced datasets, achieving a strong 9th place finish.
What Problem Does This Paper Address?
Legal reasoning tasks, especially those involving complex arguments and lengthy texts, present significant challenges for natural language processing (NLP) models. Specifically, the SemEval-2024 Task 5 on Legal Argument Reasoning in Civil Procedure requires classifying the correctness of a candidate answer given a legal topic and question. Two major hurdles exist: the input text (topic and question) often exceeds the 512-token limit of standard BERT models, and the dataset suffers from a severe imbalance between positive and negative samples, leading to models defaulting to the majority class. While Legal-BERT has shown promise in legal domains, effectively adapting it to this specific reasoning task with its inherent complexities is the core problem addressed.
Methods
The YNU-HPCC team developed a system that leverages a fine-tuned Legal-BERT model. To overcome the token length limitation, they employed two sliding window strategies:
- Sliding Window Simple (SWS): The combined topic and question are divided into segments, each classified independently, and the final prediction is an average of these segment predictions. This aims to distill the essence of the text into a single numerical value.
- Sliding Window Complex (SWC): This approach creates more comprehensive chunks by ensuring each segment includes the complete question along with relevant parts of the introduction, maintaining context. These chunks are then classified, and an aggregated output (a weighted average) is computed.
To address the data imbalance issue, two key techniques were integrated:
- R-Drop (Regularized Dropout): This method, applied to the output layer, generates two distinct predictions from the same input. A symmetrical Kullback-Leibler (KL) divergence loss is used to encourage these predictions to be consistent, effectively acting as a form of data augmentation.
- Focal Loss: Replacing the standard cross-entropy loss, focal loss is designed to down-weight easy-to-classify examples and focus training on hard negatives, which is crucial for imbalanced datasets.
The output layer of the model uses a softmax function followed by an argmax to produce the final classification.
Datasets
The primary dataset used for this task is the Legal Argument Reasoning Task in Civil Procedure shared task data set. This dataset is provided in three CSV files: train.csv (666 samples), dev.csv (84 samples), and test.csv (98 samples). The training and development sets include fields like idx, question, answer, label, analysis, complete analysis, and explanation. The test set contains idx, question, answer, and explanation. The label field indicates whether the candidate answer is correct (1) or incorrect (0).
Key Insights and Ideas
- Legal-BERT's Specialization Matters: The paper strongly suggests that Legal-BERT, pre-trained on a legal corpus, is inherently better suited for legal text than general-purpose models like BERT, RoBERTa, and DeBERTa, even when using similar techniques. This highlights the importance of domain-specific pre-training.
- Data Imbalance is a Major Hurdle: The authors found that standard models and loss functions struggled significantly with the imbalanced nature of the dataset, often predicting the majority class (0) for most inputs. This underscores the necessity of specialized techniques like R-Drop and focal loss.
- R-Drop and Focal Loss Synergy: The combination of R-Drop for regularization and focal loss for handling imbalance proved to be a powerful strategy, leading to significant improvements over baseline approaches.
- Sliding Window is Necessary but Not Sufficient: While the sliding window approach effectively addresses the token length constraint, it doesn't fully solve the reasoning problem on its own, necessitating further model enhancements.
Key Results
The YNU-HPCC team achieved a best submission F₁-score of 0.6166, ranking 9th in the SemEval-2024 Task 5 competition. The paper presents several tables comparing different models (BERT, RoBERTa, DeBERTa, Legal-BERT) with various loss functions (Cross-Entropy, Focal Loss, Dice Loss) and techniques (SCL, R-Drop). Notably, Legal-BERT consistently outperformed other models, especially when combined with Focal Loss and R-Drop. For instance, in Table 4, Legal-BERT with Focal Loss and R-Drop achieved an F₁-score of 0.6166, significantly higher than other combinations.
Limitations and Future Work
The authors acknowledge that while their sliding window approach helps with token limitations, models specifically designed for longer documents, such as Longformer or Big Bird, might offer superior efficiency for extremely long texts. They express an interest in exploring these alternative models in future work to further improve performance on long-document legal reasoning tasks. Additionally, while their system achieved good results, they note that there are still areas for improvement when compared to leading systems in the competition.
Summary updated