TL;DR
This paper details the University of Ottawa team's participation in SemEval-2023 Task 6, a challenge focused on deep learning for legal text understanding. They tackled three sub-tasks: rhetorical role prediction, legal named entity recognition, and court judgment prediction with explanation, achieving competitive results across the board by leveraging and adapting various deep learning models, including hierarchical BiLSTMs and transformer-based architectures like BERT and LegalBERT.
What Problem Does This Paper Address?
The legal domain is characterized by an overwhelming volume of complex, lengthy documents, making efficient processing and understanding a significant challenge for legal professionals. SemEval-2023 Task 6 aimed to foster a legal NLP community by addressing three specific problems:
- Rhetorical Roles Prediction (RR): Identifying the semantic function of sentences within legal texts (e.g., facts, arguments, rulings).
- Legal Named Entities Extraction (L-NER): Recognizing specific entities like parties, judges, and statutes within legal documents.
- Court Judgement Prediction with Explanation (CJPE): Predicting the outcome of a case and providing justifications for the prediction. This task seeks to develop tools that can automate aspects of legal work, improve information retrieval, and enhance the interpretability of legal AI systems.
Methods
The uOttawa team employed a range of deep learning techniques tailored to each sub-task:
- Sub-task A (RR): They experimented with a hierarchical BiLSTM-CRF model, initially using randomly initialized embeddings, and also fine-tuned transformer-based models like LegalBERT, RoBERTa-base, and InLegalBERT. Notably, the hierarchical BiLSTM-CRF with randomly initialized word embeddings yielded the best results for this sub-task.
- Sub-task B (L-NER): A BERT-based named entity recognition model was developed. They utilized the original LEGAL-BERT model for tokenization and fine-tuned it using the
BertForTokenClassificationclass. The data was preprocessed into the BIO format, and label alignment was crucial due to tokenization. - Sub-task C (CJPE): For predicting court judgment outcomes (CJP), they fine-tuned transformer models like LegalBERT and CaseLawBERT, using them as classifiers or as encoders for subsequent BiGRU and RCNN classifiers. For the explanation part (CJPE), they adapted a method involving masking chunk embeddings to calculate explainability scores and then used sentence clustering (K-means) to select explanatory sentences.
Datasets
The team utilized datasets provided by the LegalEval 2023 organizers for each sub-task:
- Sub-task A: Indian Court Judgements, annotated with 13 predefined rhetorical roles. The dataset was split into training, development, and test sets.
- Sub-task B: A dataset for legal named entity recognition, also provided by the organizers, split into training, development, and test sets. This dataset included preambles and judgments.
- Sub-task C: The ILDC dataset, a subset of cases from the Supreme Court of India, comprising ILDC-single and ILDC-multi documents. This dataset was used for both judgment prediction (C-1) and explanation generation (C-2).
Key Insights and Ideas
- Hierarchical BiLSTM-CRF's surprising effectiveness: For rhetorical role prediction, a hierarchical BiLSTM-CRF model with randomly initialized embeddings outperformed transformer-based models, suggesting that for certain legal NLP tasks, simpler architectures can be highly competitive.
- Domain-specific transformers are powerful but require careful adaptation: While models like LegalBERT and CaseLawBERT are pre-trained on legal texts, their effectiveness on specific legal systems (like Indian law) still relies on fine-tuning and careful adaptation.
- The challenge of legal text length: The paper notes that models struggled with longer test texts, indicating that handling long-range dependencies remains a key challenge in legal NLP.
- Explainability is difficult: The results for the explanation sub-task (C-2) were modest, highlighting the inherent difficulty in automatically generating accurate and meaningful explanations for legal decisions.
Key Results
- Sub-task A (RR): Achieved a competitive micro F1-score, with the hierarchical BiLSTM-CRF model with randomly initialized embeddings performing best at 0.74.
- Sub-task B (L-NER): Obtained an F1-score of 0.87, ranking sixth overall in the shared task.
- Sub-task C-1 (CJP): Placed third among 11 teams, with their best model, CaseLawBERT + BiGRU, achieving an F1-score of 0.68.
- Sub-task C-2 (CJPE): Ranked 10th, with their best approach achieving a ROUGE-2 score of 0.040, which was significantly lower than the baseline and top-performing systems.
Limitations and Future Work
- Limitations:
- The validation results were better than test results, suggesting models struggled with the longer texts in the test set.
- The explanation sub-task yielded low scores, indicating the difficulty of generating accurate explanations.
- The team noted that some transformer models showed low F1-scores on the test set for Sub-task A, despite good performance on the validation set.
- Future Work:
- Investigate combining legal tasks, such as Legal-NER and CJPE, to see if entity identification can improve judgment interpretation.
- Explore emerging AI-powered solutions like legal prompting.
- Further research into handling longer legal documents and improving the explainability of legal AI models.
Summary updated