TeamUnibo at SemEval-2023 Task 6: A transformer based approach to Rhetorical Roles prediction and NER in Legal Texts

Yuri Noviello, Enrico Pallotta, Flavio Pinzarrone, Giuseppe TanziACL 2023

TL;DR

This paper presents transformer-based approaches for two crucial legal NLP tasks: Rhetorical Role prediction and Named Entity Recognition (NER). By leveraging context-aware models and specialized legal pre-trained transformers like InLegalBERT, the team achieved competitive results, highlighting the effectiveness of these methods for navigating the complexities of legal documents.

What Problem Does This Paper Address?

The legal domain presents unique challenges for Natural Language Processing (NLP). Legal texts are often lengthy, unstructured, and filled with specialized jargon, making them difficult for standard NLP models to process. This paper tackles two specific problems within this domain:

  1. Rhetorical Role Prediction (RRP): Identifying the function of each sentence within a legal document (e.g., preamble, fact, argument, ratio). This is a foundational step for many legal AI applications like summarization and judgment prediction.
  2. Named Entity Recognition (NER): Extracting and classifying key entities within legal texts, such as names of parties, courts, and statutes. This is essential for building knowledge graphs and improving information retrieval.

The LegalEval challenge aims to push the boundaries of NLP in this domain, and this work contributes by developing specialized models to address these specific tasks.

Methods

The team employed transformer-based architectures, a state-of-the-art approach in NLP, and adapted them for the legal domain.

For Rhetorical Role Prediction (Task A):

  • Baselines: Standard transformer models (BERT, RoBERTa) were used for single-sentence classification.
  • Context-Aware Approach: Recognizing that sentences in legal documents often have contextual dependencies, they introduced a "context-aware" method. This involved enriching sentence representations by including tokens from surrounding sentences. This was achieved by:
  • Adding BiLSTM layers to the transformer embeddings.
  • Extending the input to the transformer with a context window.
  • Legal-Specific Transformer: They utilized InLegalBERT, a transformer pre-trained on Indian legal documents, which proved to be highly effective.

For Named Entity Recognition (Task B):

  • Baseline: A standard transformer model with a simple linear layer as a classification head.
  • Custom Model: They explored more complex architectures by stacking a BiLSTM and a CRF (Conditional Random Field) layer on top of the transformer. This combination is known to improve sequence labeling performance by considering label dependencies.
  • Sliding Window: To handle the long sequences in legal documents that exceed transformer input limits (often 512 tokens), they implemented a sliding window approach for both training and inference. This allows the model to process longer texts by breaking them into overlapping segments.

Datasets

The research utilized datasets provided by the LegalEval challenge:

  • Task A (Rhetorical Roles Prediction): The Corpus for Automatic Structuring of Legal Documents (Kalamkar et al., 2022b), containing over 40,000 sentences annotated with 12 rhetorical roles. The challenge provided smaller training and development sets, and an additional unannotated test set for evaluation.
  • Task B (Legal Named Entity Recognition): The Legal Named Entity Recognition Corpus (Kalamkar et al., 2022a), comprising over 16,000 sentences from Indian court judgments annotated with 14 entity types. Similar to Task A, smaller training and development sets were provided for the challenge.

Key Insights and Ideas

  • Context is King for RRP: Simply classifying sentences in isolation is insufficient for legal texts. Incorporating surrounding sentence context significantly boosts performance in Rhetorical Role Prediction.
  • Legal-Specific Pre-training Matters: Transformers pre-trained on legal corpora (like InLegalBERT) outperform general-domain models, demonstrating the value of domain-specific knowledge.
  • NER Benefits from Sequence Modeling: While simple transformer heads work, adding BiLSTM and CRF layers can further enhance NER performance by capturing sequential label dependencies.
  • Handling Long Documents is Crucial: Legal documents' length necessitates techniques like sliding windows to avoid information loss due to input truncation.
  • Class Imbalance is a Challenge: Legal documents exhibit inherent class imbalance (e.g., many "ANALYSIS" sentences, few "PRE_NOT_RELIED"), making it difficult for models to perform well on minority classes.

Key Results

Task A - Rhetorical Roles Prediction:

  • The context-aware model using InLegalBERT achieved a Micro-F1 score of 81.12% on the submission data, securing the 8th position in the leaderboard.
  • Context-aware models (M1 and M2) showed a 10% improvement in Micro-F1 scores compared to single-sentence baseline models.

Task B - Legal Named Entity Recognition:

  • The custom model (XLNet + BiLSTM + CRF) achieved a Macro-F1 score of 87.43% on the submission data.
  • XLNet was found to be the best transformer backbone for NER in this study.
  • The custom model (CM2) outperformed the baseline XLNet model (BM3) by 1.39% (87.43% vs 86.04% F1).

Limitations and Future Work

  • RRP Minority Class Performance: The model struggles with minority classes in RRP, likely due to class imbalance. The authors suggest a multi-label setup as a potential solution.
  • NER Document-Level Context: The current NER model processes sentences independently. Future work could explore incorporating document-level context, perhaps through post-processing steps or by modifying the architecture to handle longer sequences.
  • NER Error Analysis: Errors in NER were observed due to capitalization (affecting organization recognition) and the confusion between entities like precedents and case numbers. The authors suggest addressing these by incorporating document-level context or using a multi-label approach.
  • Computational Cost: Context-aware models, while effective, increase input size and thus training/inference times, which was a constraint on GPU memory.

Summary updated

BibTeX
@inproceedings{noviello2023teamunibo,
  author = {Yuri Noviello and Enrico Pallotta and Flavio Pinzarrone and Giuseppe Tanzi},
  title = {{TeamUnibo at SemEval-2023 Task 6: A transformer based approach to Rhetorical Roles prediction and NER in Legal Texts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure