Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation

Shubham Kumar Nigam, Aniket Deroy, Noel Shallum, Ayush Kumar Mishra, Anup Roy, Shubham Kumar Mishra, Arnab Bhattacharya, Saptarshi Ghosh, Kripabandhu Ghosh

This paper presents the work of Team Nonet at SemEval-2023 Task 6, focusing on the automation of legal analysis within the Indian judicial system. By combining transformer-based architectures with domain-specific heuristics, the authors tackled three distinct challenges: identifying legal entities, predicting case outcomes, and extracting textual explanations for those predictions.

TL;DR

Team Nonet developed a suite of NLP tools for the Indian legal domain, achieving a 1st place ranking in the Court Judgment Prediction with Explanation (CJPE) subtask. Their approach demonstrates that while complex transformer ensembles excel at entity recognition, simple heuristics focusing on the "tail" of legal documents are surprisingly effective for explaining judicial decisions.

What Problem Does This Paper Address?

The Indian legal system faces a massive backlog of cases, exacerbated by a shortage of judicial resources and a large population. Automating parts of the legal workflow—such as identifying key actors (NER) or predicting outcomes—could significantly reduce this burden.

However, legal text is notoriously difficult for standard NLP models. It contains domain-specific entities (e.g., "statutes," "precedents," "petitioners") that general-purpose models miss. Furthermore, legal documents are often extremely long, exceeding the token limits of standard transformer models like BERT, and the reasoning behind a decision is often buried deep within the text.

Methods

The authors participated in three subtasks, employing a mix of standard and custom architectures:

The team used a fusion approach combining two models:

  • Modified spaCy Pipeline: A RoBERTa transformer fine-tuned with domain-specific Law2Vec embeddings.
  • Fine-Tuned BERT-CRF: A Legal-BERT model with a token-level classifier and a Linear-Chain Conditional Random Field (CRF) layer. To make the model more robust, they modified the embedding layer to include Part-of-Speech (POS) tags, as most legal entities are nouns.
Figure 2: Modified BERT Embedding Layer

To handle long documents, the authors experimented with two strategies:

  • Tail-Truncation: Passing only the last 510 tokens of a judgment to transformers like XLNet and RoBERTa, based on the observation that judicial decisions are usually summarized at the end.
  • Hierarchical Transformers: Segmenting the document into chunks and using a sequential model (BiGRU + attention) to process the chunk embeddings.

3. Court Judgment Prediction with Explanation (CJPE)

For the explanation task, the team opted for a simple, intuitive approach over complex explainability methods like Integrated Gradients. They used keyword-based pattern matching to identify the decision (e.g., "dismissed," "allowed") and extracted continuous text spans of varying lengths from the end of the document to serve as the explanation.

Figure 4: Court Judgment Prediction and Explanation (Task C) schematic diagram

Datasets

The study utilized datasets provided by the SemEval-2023 organizers:

  • L-NER Dataset: Divided into "Preamble" (names of parties, court, lawyers) and "Judgment" (precedents, provisions, witnesses). It included 13 tags for judgments and 5 for preambles.
  • ILDC (Indian Legal Document Corpus): Used for LJP, containing thousands of single and multi-judge case documents.
  • CJPE Dataset: A smaller set of 50 legal judgments used for the explanation subtask.
Figure 5: Judgement Tag Distribution

Key Insights and Ideas

  • Model Fusion for NER: Merging the outputs of the spaCy and BERT-CRF models improved accuracy by 2% compared to using spaCy alone. The fusion logic merged intersecting spans and took the union of non-overlapping ones.
  • The "Tail" Heuristic: In the Indian legal context, the most syntactically and semantically important information regarding the final decision typically occurs at the end of the document. Models focusing on the last 510 tokens often outperformed complex hierarchical models.
  • Simplicity vs. Sophistication: The team won the CJPE subtask using a simple "last 300 words" heuristic. They argue that sophisticated explainability methods (like Integrated Gradients) fail when the underlying prediction model's accuracy is low, as the gradients become unreliable.

Key Results

  • CJPE (Task C2): Ranked 1st on the leaderboard with a Macro F1 of 0.5417 and a ROUGE-2 score of 0.0473. The best results were achieved by extracting the last 300 words of the judgment.
  • LJP (Task C1): Ranked 11th, achieving a Macro F1 of 0.5287 using a pre-trained XLNet-large model.
  • L-NER (Task B): Ranked 15th with an F1-score of 0.5532 on the test set. While validation scores were high (weighted average F1 ~84%), there was a significant drop on the hidden test set.

Limitations and Future Work

The authors noted a significant performance gap between their validation and test results for NER, but were unable to perform a detailed error analysis as the labeled test data was not released.

For future work, they plan to explore joint optimization frameworks that incorporate rhetorical roles (e.g., identifying which sentences are "Facts" vs. "Statutes") alongside sentence position to create more precise span detection models for legal explanations.

Summary updated

Abstract

This paper describes our submission to the SemEval-2023 for Task 6 on LegalEval: Understanding Legal Texts. Our submission concentrated on three subtasks: Legal Named Entity Recognition (L-NER) for Task-B, Legal Judgment Prediction (LJP) for Task-C1, and Court Judgment Prediction with Explanation (CJPE) for Task-C2. We conducted various experiments on these subtasks and presented the results in detail, including data statistics and methodology. It is worth noting that legal tasks, such as those tackled in this research, have been gaining importance due to the increasing need to automate legal analysis and support. Our team obtained competitive rankings of 15$^{th}$, 11$^{th}$, and 1$^{st}$ in Task-B, Task-C1, and Task-C2, respectively, as reported on the leaderboard.
BibTeX
@inproceedings{nigam2023nonet,
  author = {Shubham Kumar Nigam and Aniket Deroy and Noel Shallum and Ayush Kumar Mishra and Anup Roy and Shubham Kumar Mishra and Arnab Bhattacharya and Saptarshi Ghosh and Kripabandhu Ghosh},
  title = {{Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure