Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation

Shubham Kumar Nigam, Aniket Deroy, Noel Shallum, Ayush Kumar Mishra, Anup Roy, Shubham Kumar Mishra, Arnab Bhattacharya, Saptarshi Ghosh, Kripabandhu Ghosh

TL;DR

This paper presents the Nonet team's approach to SemEval-2023 Task 6, focusing on automating Indian legal text analysis. They tackled three subtasks: Legal Named Entity Recognition (L-NER), Legal Judgment Prediction (LJP), and Court Judgment Prediction with Explanation (CJPE), achieving impressive rankings, including first place in CJPE. Their work highlights the potential of combining pre-trained transformer models with domain-specific techniques for legal NLP.

What Problem Does This Paper Address?

The Indian legal system, like many others, faces challenges with slow processing times and limited accessibility due to a large volume of cases and a shortage of judicial resources. SemEval-2023 Task 6 aimed to address these issues by developing systems to automate various aspects of legal text analysis. Specifically, the task focused on:

  • Legal Named Entity Recognition (L-NER): Identifying crucial entities within legal documents (e.g., petitioner, respondent, judge, statute) that are often not recognized by general-purpose NER systems. This is particularly challenging due to the unique terminology and structure of Indian legal texts.
  • Legal Judgment Prediction (LJP): Automatically predicting the outcome of a legal judgment (e.g., favoring the appellant or defendant). This has the potential to significantly reduce the burden on the judicial system by automating preliminary case analysis.
  • Court Judgment Prediction with Explanation (CJPE): Not only predicting the outcome but also identifying the specific text spans within the judgment that provide the reasoning for that decision. This adds a crucial layer of transparency to the automated legal process.

Methods

The Nonet team employed a combination of established and novel techniques across the three subtasks:

  • Legal Named Entity Recognition (L-NER):
    • SpaCy Model: They fine-tuned a RoBERTa transformer model and integrated domain-specific embeddings from Law2Vec into a modified SpaCy pipeline. This approach leverages the contextual understanding of transformers and specialized legal vocabulary.
  • Fine-Tuned BERT Model: A BERT model with a token-level classifier and a Linear-Chain CRF was used. They incorporated Part-of-Speech (POS) tags as an additional feature to improve the accuracy of entity recognition, especially for subword tokens.
  • Model Fusion: For L-NER, they fused the outputs of the SpaCy and BERT-CRF models by merging overlapping entity predictions to achieve better overall performance.
  • Legal Judgment Prediction (LJP):
    • Transformer Models: They experimented with pre-trained transformers like XLNet and RoBERTa, as well as legal-specific BERT models (LegalBERT, InlegalBERT, IncaseLawBERT). Due to transformer input length limitations, they focused on the last 512 tokens of the judgment, hypothesizing that crucial information resides there.
  • Hierarchical Transformers: They also explored a hierarchical transformer architecture by dividing documents into chunks and using the [CLS] representation as input to sequential models. However, this approach did not yield better results than standard transformers.
  • Court Judgment Prediction with Explanation (CJPE):
    • Keyword-Based Matching: For identifying the decision (favoring appellant or not), they used a simple keyword-matching approach, looking for terms like "dispose of," "accept," "dismiss," and "reject."
    • Span Selection: For explanations, they adopted an intuitive approach of selecting continuous text spans from the end of the legal judgment. They experimented with various span lengths (e.g., last 300 words) to find the most effective ones.

Datasets

The paper utilized datasets provided by the SemEval-2023 Task 6 organizers. These datasets were in English and covered the Indian legal context.

  • L-NER: The dataset was divided into two sections: "Preamble" (containing party names, court, lawyers) and "Judgment" (containing court, case number, precedent, etc.).
  • LJP: The dataset consisted of two sets: ILDC_single and ILDC_multi, with a significant number of training, validation, and test documents.
  • CJPE: This subtask used a smaller dataset of 50 legal court judgments.

Key Insights and Ideas

  • Domain-Specific Embeddings Matter: Integrating Law2Vec embeddings significantly enhanced the contextual understanding of legal terms for the L-NER task.
  • End of Document is Key: For LJP, focusing on the latter parts of the legal judgment proved effective, aligning with the hypothesis that crucial decision-making information is often found towards the end.
  • Simplicity Can Be Effective: For CJPE, a straightforward keyword-based approach for prediction and selecting text spans from the end of the document outperformed more complex explainability methods, especially when the underlying prediction model was not highly accurate.
  • Model Fusion Boosts NER: Combining predictions from different NER models (SpaCy and BERT-CRF) led to improved accuracy by leveraging their complementary strengths.
  • POS Tags Aid BERT: Incorporating Part-of-Speech tags as an additional feature for BERT improved its ability to recognize named entities.

Key Results

The Nonet team achieved competitive rankings across all subtasks:

  • L-NER: Ranked 15th with an F1-Score of 0.5532 on the test dataset.
  • LJP: Ranked 11th with a Macro F1 score of 0.5287.
  • CJPE: Achieved a remarkable 1st place ranking. The best explanation span length was found to be the last 300 words, covering approximately 19.26% of the document.

Limitations and Future Work

  • Lack of Labeled Test Data: A significant limitation acknowledged is the absence of a labeled test dataset for L-NER, preventing a comprehensive error analysis on the test set. The authors plan to conduct this analysis once the gold standard labels become available.
  • Hierarchical Transformer Performance: The hierarchical transformer models did not outperform standard transformers for LJP, possibly due to insufficient data to effectively pass embedding information to sequential models.
  • Explainability Method Limitations: The authors note that sophisticated explainability methods like Integrated Gradients might not perform well if the underlying prediction model has low F1 scores.
  • Future Work: The team plans to explore joint optimization frameworks that combine rhetorical role information and sentence position for more accurate span detection in CJPE. They also aim to investigate the effectiveness of their proposed models and techniques for other legal domain tasks.

Summary updated

Abstract

This paper describes our submission to the SemEval-2023 for Task 6 on LegalEval: Understanding Legal Texts. Our submission concentrated on three subtasks: Legal Named Entity Recognition (L-NER) for Task-B, Legal Judgment Prediction (LJP) for Task-C1, and Court Judgment Prediction with Explanation (CJPE) for Task-C2. We conducted various experiments on these subtasks and presented the results in detail, including data statistics and methodology. It is worth noting that legal tasks, such as those tackled in this research, have been gaining importance due to the increasing need to automate legal analysis and support. Our team obtained competitive rankings of 15$^{th}$, 11$^{th}$, and 1$^{st}$ in Task-B, Task-C1, and Task-C2, respectively, as reported on the leaderboard.
BibTeX
@inproceedings{nigam2023nonet,
  author = {Shubham Kumar Nigam and Aniket Deroy and Noel Shallum and Ayush Kumar Mishra and Anup Roy and Shubham Kumar Mishra and Arnab Bhattacharya and Saptarshi Ghosh and Kripabandhu Ghosh},
  title = {{Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure