PoliToHFI at SemEval-2023 Task 6: Leveraging Entity-Aware and Hierarchical Transformers For Legal Entity Recognition and Court Judgment Prediction

Irene Benedetto, Alkis Koudounas, Lorenzo Vaiani, Eliana Pastor, Elena Baralis, Luca Cagliero, Francesco TarasconiACL 2023

TL;DR

This paper tackles two crucial legal AI tasks: Legal Named Entity Recognition (L-NER) and Court Judgment Prediction with Explanation (CJPE). The authors propose novel transformer-based approaches, including an entity-aware attention mechanism for L-NER and hierarchical models with explainability enhancements for CJPE, achieving competitive results and offering insights into domain-specific pre-training and explanation quality.

What Problem Does This Paper Address?

The legal domain presents unique challenges for Natural Language Processing (NLP) due to its specialized language, complex structures, and the need for high accuracy and interpretability. Specifically, identifying key entities within legal documents (L-NER) and predicting the outcomes of court cases while providing justifications (CJPE) are critical for legal professionals. Existing NLP models often struggle with the nuances of legal text, and the interpretability of predictive models is paramount for trust and adoption in this high-stakes field. This paper aims to improve both entity recognition and prediction/explanation capabilities within the legal context.

Methods

The paper introduces several key methodological advancements:

  • Legal Named Entity Recognition (L-NER): The authors explore transformer-based models, notably incorporating an entity-aware attention mechanism (inspired by LUKE). This mechanism allows the model to not only attend to token sequences but also to specific domain entities, aiming to improve the recognition of legal entities. They also investigate the impact of domain-specific pre-training versus general-purpose pre-training followed by fine-tuning.
  • Court Judgment Prediction (CJP): For predicting case outcomes, the approach utilizes hierarchical transformer-based models. These models process documents sentence by sentence, building up representations hierarchically. They experiment with various transformer encoders (e.g., RoBERTa, Legal BERT, Longformer) and analyze the impact of document type (single vs. multi-petition cases).
  • Court Judgment Prediction with Explanation (CJPE): This task builds upon the CJP models. The core innovation here is the use of post-hoc explanation methods (like Gradient, Integrated Gradient, and Leave-One-Out) adapted to provide sentence-level importance scores. Crucially, they propose a novel approach to enhance these explanations by leveraging Named Entity Recognition (NER) tagging. The intuition is that sentences containing legal entities are more likely to be relevant to the prediction, and this information is used to "boost" the explanation scores.

Datasets

The paper utilizes existing datasets for their tasks:

  • L-NER: The authors mention using datasets like the European Court of Human Rights (ECHR) and Legal BERT corpora for fine-tuning and evaluation. They also refer to publicly available legal NER datasets.
  • CJP/CJPE: The primary dataset mentioned is the Indian Legal Documents Corpus (ILDC), which contains court judgments. They differentiate between single-type and multi-type datasets within ILDC, where multi-type refers to cases with multiple petitions.

Key Insights and Ideas

  • Domain-Specific Pre-training Matters: For L-NER, fine-tuning models on legal text (like ECHR) consistently outperforms models fine-tuned on general text, highlighting the importance of domain-specific knowledge.
  • Entity-Aware Attention Boosts L-NER: The entity-aware attention mechanism shows promise in improving L-NER performance by allowing models to focus on domain-specific entities.
  • Hierarchical Models for Long Legal Texts: Hierarchical transformer architectures are effective for handling the long documents common in legal settings for CJP.
  • NER-Enhanced Explanations: Leveraging NER tags to boost sentence-level explanations in CJPE is a promising direction. The intuition that legal entities are key to understanding judgments is empirically supported, leading to more focused and potentially more meaningful explanations.
  • Conciseness in Explanations: Shorter explanations (e.g., 25-30% of sentences) tend to yield better evaluation scores, suggesting that brevity aids in conveying the core reasoning.

Key Results

  • L-NER: The BERT-base model fine-tuned on ECHR achieved the best test set F1 score of 83.18%, closely followed by Legal BERT-base (83.09%). This reinforces the benefit of domain-specific fine-tuning.
  • CJP: The hierarchical models achieved performance comparable to baseline methods, with RoBERTa-large and Legal BERT showing strong results for both single and multi-type documents. The study noted a roughly 10% drop in performance on the test set compared to the development set.
  • CJPE: The proposed CJPE approach, using NER-boosted explanations, showed promising results in producing concise yet informative explanations. While quantitative results for CJPE quality are presented using various metrics (ROUGE, BLEU, etc.), the paper highlights that the NER boosting did not show a significant impact across all metrics when considering 40% of sentences as explanations, but it was more effective for shorter explanations. The model achieved an F1-score of 45.25% on the CJPE task.
  • Ranking: The authors participated in SemEval-2023 Task 6 and ranked 11th out of 17 for L-NER, 4th out of 11 for CJP, and 5th out of 11 for CJPE.

Limitations and Future Work

  • Dataset Limitations: The paper notes that the unknown typology of test documents for CJP could not be effectively managed through ensembling, leading to performance degradation on the test set.
  • Explanation Quality: While the study assesses explanation quality, it acknowledges the need for more comprehensive assessments of plausibility and faithfulness, especially compared to human expert rationales.
  • Subgroup Performance: The authors suggest evaluating CJP performance at the subgroup level to identify potential biases or performance disparities across different data segments.
  • Future Work: The authors plan to explore the applicability of entity-aware transformers to other legal AI tasks, such as relation extraction, to better capture inter-dependencies between entities. They also aim for a more comprehensive assessment of explanation plausibility and faithfulness.

Summary updated

BibTeX
@inproceedings{benedetto2023politohfi,
  author = {Irene Benedetto and Alkis Koudounas and Lorenzo Vaiani and Eliana Pastor and Elena Baralis and Luca Cagliero and Francesco Tarasconi},
  title = {{PoliToHFI at SemEval-2023 Task 6: Leveraging Entity-Aware and Hierarchical Transformers For Legal Entity Recognition and Court Judgment Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure