UO-LouTAL at SemEval-2023 Task 6: Lightweight Systems for Legal Processing

Sébastien Bosch, Louis Estève, Joanne Loo, Anne-Lyse MinardACL 2023

TL;DR

This paper presents lightweight and interpretable systems for two sub-tasks of the LegalEval 2023 challenge: Legal Named Entity Recognition (L-NER) and Rhetorical Role (RR) prediction. The authors demonstrate that even with resource-constrained models like CRFs and Bag-of-Words, competitive performance can be achieved, highlighting the trade-offs between model complexity, interpretability, and computational cost in legal NLP.

What Problem Does This Paper Address?

The legal domain is awash in documents, making manual information retrieval and processing a time-consuming and labor-intensive task. Automated systems can significantly alleviate this burden, freeing up legal professionals for more complex cognitive tasks. However, many state-of-the-art NLP models, particularly transformer-based architectures, are computationally expensive, require specialized hardware, and can be opaque, hindering their widespread adoption and trustworthiness in the legal field. This paper tackles the challenge of developing "lightweight" and "interpretable" systems for legal text processing, aiming to balance performance with practical deployability and user trust.

Methods

The authors focused on two sub-tasks:

  • Legal Named Entity Recognition (L-NER): They employed a Conditional Random Field (CRF) model, a standard and interpretable approach for sequence labeling. To enhance performance, they augmented the CRF with post-processing rules for specific named entity types.
  • Rhetorical Role (RR) Prediction: This task involves classifying sentences within legal documents into predefined rhetorical roles. Two main approaches were explored:
  • Bag-of-Words (BoW): A traditional text classification method using TF-IDF features and n-grams, combined with a Logistic Regression classifier.
  • Sentence-Transformers: A deep learning approach utilizing pre-trained sentence embeddings (all-MiniLM-L6-v2) fed into a K-Neighbours classifier.

For both sub-tasks, a key aspect was the integration of L-NER outputs into the RR systems. Additionally, rule-based post-processing was applied to refine the predictions of both L-NER and RR systems. The definition of "lightweight" here means training within hours on a single machine without specialized hardware.

Datasets

The paper utilizes datasets provided by the LegalEval 2023 organizing team for both the L-NER and RR sub-tasks. These datasets are divided into TRAIN, DEV, and TEST sets.

  • L-NER: The dataset consists of legal documents split into "preamble" and "judgement" sections.
  • RR: The dataset comprises Indian court judgments, characterized by their substantial length. The rhetorical roles are identified at the sentence level.

No new benchmarks were introduced; the focus was on participating in the existing LegalEval 2023 task.

Key Insights and Ideas

  • Lightweight is Achievable: The study demonstrates that even with simpler models like CRFs and BoW, reasonable performance can be achieved in specialized legal NLP tasks, challenging the notion that only massive deep learning models can succeed.
  • Interpretability Matters: The authors explicitly prioritized interpretability, noting that CRFs offer inherent explainability through feature weights, which is crucial for building trust with legal professionals.
  • Synergy Between Tasks: Integrating L-NER outputs as features for RR prediction showed a positive impact, suggesting that a pipeline approach where NER precedes rhetorical role identification can be beneficial.
  • Post-processing Trade-offs: While post-processing rules (especially for L-NER) could improve recall, they sometimes led to a detriment in precision, highlighting the need for careful rule design and evaluation.
  • Data Imbalance Challenges: In the RR task, the performance was significantly affected by class imbalance, with well-represented classes performing much better than rare ones.

Key Results

  • L-NER: The CRF system achieved a Macro F1 score of 0.74 on the DEV set and 0.64 on the TEST set. The best performance on the TEST set was achieved without post-processing.
  • RR: The best-performing system was the Bag-of-Words approach with both L-NER pre-processing and rule-based post-processing, achieving a Macro F1 score of 0.41 on the DEV set. On the TEST set, the Bag-of-Words system with L-NER pre-processing achieved 0.568.
  • The authors' systems placed 25th out of 27 for RR and 14th out of 17 for L-NER in the LegalEval 2023 competition.

Limitations and Future Work

  • Generalization: The significant drop in L-NER performance from the DEV set (0.74) to the TEST set (0.64) suggests potential issues with generalization across different data distributions.
  • Differentiating Similar Classes: The CRF model struggled to differentiate between similar person-related entity types (e.g., PETITIONER vs. RESPONDENT) in L-NER.
  • Interpretability vs. Performance: While lightweight systems were prioritized, the authors acknowledge that fine-tuning sentence-transformer models on legal data could improve performance but would likely compromise their lightweight and interpretable nature.
  • Future Work: For L-NER, incorporating an LSTM before the CRF is suggested to potentially boost performance while maintaining a degree of lightness. For RR, the authors recommend capitalizing on the synergy between the two sub-tasks by deploying them in sequence. Addressing class imbalance in the RR task is also highlighted as a crucial area for improvement.

Summary updated

BibTeX
@inproceedings{bosch2023uo,
  author = {Sébastien Bosch and Louis Estève and Joanne Loo and Anne-Lyse Minard},
  title = {{UO-LouTAL at SemEval-2023 Task 6: Lightweight Systems for Legal Processing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure