TL;DR
This paper explores contrastive learning (SetFit) for legal text classification, demonstrating it outperforms traditional finetuning on scarce, imbalanced legal datasets. Crucially, it shows SetFit models rely more heavily on legally informative features, enhancing trustworthiness.
What Problem Does This Paper Address?
Legal text classification research faces a significant hurdle: the scarcity of publicly available, high-quality, and balanced legal datasets. This data imbalance and scarcity can severely impact the performance and reliability of machine learning models, especially when adapting pre-trained models to specific downstream tasks. While few-shot learning methods offer a potential solution, their effectiveness and the interpretability of their decisions in the legal domain remain areas for exploration.
Methods
The core of this research lies in comparing two finetuning objectives for the LegalBERT model on legal provision classification:
- Vanilla Finetuning: A standard approach where the pre-trained LegalBERT model is directly finetuned on the target dataset.
- SetFit (Sentence Transformer Finetuning): A contrastive learning framework designed for few-shot learning. SetFit first uses contrastive learning to generate positive and negative sentence pairs from the existing labeled data, effectively augmenting the training set. This is followed by training a classification head on the embeddings generated by the finetuned Sentence Transformer.
To understand why the models make certain predictions, the authors employ LIME (Local Interpretable Model-agnostic Explanations). LIME helps identify which words (features) contribute most positively or negatively to a classification decision, allowing for an analysis of the model's reliance on legally relevant terms.
Datasets
- LEDGAR Dataset: The primary dataset used is the publicly available LEDGAR provision classification dataset, containing approximately 60,000 training samples across 100 provision labels.
- Custom Test Datasets:
- For the unbalanced data scenario, a test set was created by randomly selecting 25 samples per label from the original corpus (approx. 2,500 samples).
- For the balanced data scenario, a dedicated test dataset was created by selecting 25 samples per label for 32 classes (total 800 samples).
- Balanced LEDGAR Dataset: A new balanced dataset was created by selecting the 32 most frequent labels from LEDGAR. For labels with over 1000 samples, they were downsampled to 1000. For labels with fewer than 1000 samples, data was upsampled by crawling from LawInsider. This resulted in a dataset with 32 classes, each having 1000 provisions.
Key Insights and Ideas
- Contrastive Learning for Legal Data: SetFit's contrastive learning approach significantly boosts performance on legal text classification, especially when data is scarce. This suggests that explicitly teaching the model to distinguish between similar and dissimilar legal texts is highly beneficial.
- Feature Importance in Legal AI: The LIME analysis reveals that SetFit models tend to rely more heavily on legally informative features for their decisions, contributing to greater trustworthiness. This is a critical insight for deploying AI in high-stakes domains like law.
- Efficiency of SetFit: SetFit achieves comparable or better performance than vanilla finetuning while using a fraction of the training data, highlighting its efficiency for low-resource scenarios.
- Data Imbalance Matters: The difference in performance between SetFit and vanilla finetuning is more pronounced when dealing with imbalanced datasets, as indicated by the larger gap in Weighted-F1 scores.
Key Results
- F1-Scores: On the original LEDGAR dataset, SetFit consistently outperformed vanilla finetuning across various sample sizes. For instance, with 400 training samples, SetFit achieved a Micro-F1 of 0.6565, compared to 0.5805 for vanilla finetuning with 4933 samples. Even with more data, SetFit maintained a competitive edge.
- Accuracy: Accuracy comparisons showed comparable results between SetFit and vanilla models, particularly in small data scenarios. However, SetFit's ability to achieve this with significantly less data is a key takeaway.
- LIME Analysis:
- For the "Adjustments" provision, SetFit highlighted features like "adjustments," "shares," and "dividend" more strongly than vanilla finetuning.
- For the "Authority" provision, SetFit emphasized terms like "authority," "execute," and "binding."
- In contrastive settings, negative features (words that decrease the probability of a label) were also more pronounced, sometimes including less informative terms, but the overall trend suggested a stronger reliance on legally relevant features.
Limitations and Future Work
- Hyperparameter Tuning: The paper acknowledges that further investigation into SetFit's hyperparameters, especially on legal data, could potentially enhance its capabilities.
- Explainability Tools: The authors suggest exploring other explainability tools like SHAP or GRAD-CAM for a more comprehensive feature analysis.
- Domain Expert Validation: A crucial future step is to validate the appropriateness of identified positive and negative features with legal domain experts to confirm their true legal informativeness and impact on model decisions.
- Broader Legal Tasks: While this study focused on provision classification, future work could explore SetFit's effectiveness on other legal NLP tasks.
Summary updated