NLP-Titan at SemEval-2023 Task 6: Identification of Rhetorical Roles Using Sequential Sentence Classification

Harsh Kataria, Ambuje GuptaACL 2023

TL;DR

This paper tackles the challenge of automatically identifying rhetorical roles within lengthy legal documents, a crucial step for efficient legal text processing. The authors propose a novel two-stage approach combining a Hierarchical Sequential Labeling Network (HSLN) for initial classification and SetFit for fine-grained distinctions, achieving a state-of-the-art F1 score of 0.8301 on the SemEval-2023 Task 6 dataset.

What Problem Does This Paper Address?

Legal documents, particularly in developing countries like India, are notoriously long, complex, and unstructured, leading to a massive backlog of pending cases. Extracting meaningful information and understanding the argumentative structure within these documents is a significant hurdle for researchers, practitioners, and legal professionals. Traditional NLP models often struggle with the sheer length and specialized jargon of legal texts. This paper aims to bridge this gap by developing a system that can automatically segment legal documents into distinct rhetorical roles, making them more digestible and processable.

Methods

The core of the proposed solution is a two-stage classification pipeline.

  • Stage 1: Hierarchical Sequential Labeling Network (HSLN): This stage uses a Hierarchical Sequential Labeling Network (HSLN) architecture, adapted from previous work on scientific articles. The HSLN processes sentences sequentially, first encoding words into embeddings using BERT Base Uncased. These word embeddings are then fed into a Bi-LSTM to capture sentence-level context, followed by another Bi-LSTM to capture inter-sentence context. Finally, a Conditional Random Field (CRF) layer predicts the rhetorical role for each sentence. To handle the extensive nature of legal documents, the authors chose HSLN for its ability to process documents of any length. They also froze BERT parameters to reduce computational complexity.
  • Stage 2: SetFit for Fine-Grained Classification: Before the HSLN stage, the authors merged 13 original rhetorical roles into 10 broader categories to simplify the initial classification. After the HSLN predicts these merged categories, a second stage using SetFit is employed. SetFit, a few-shot learning model, is used to classify three specific merged categories (AST, REL, RXC) back into their original, more granular roles. This two-stage approach, particularly the merging and then fine-grained classification, is key to their improved performance.

Datasets

The study utilizes the dataset provided by the SemEval-2023 Task 6 organizers. This dataset consists of annotated legal documents from Indian legal cases, including those from the Supreme Court of India, high courts, and district-level courts. The dataset is split into training (247 cases) and development (30 cases) sets. The cases belong to either 'Tax' or 'Criminal' categories. The paper also references the "Legal-NLP-EkStep's rhetorical role baseline github repository" for definitions of rhetorical roles.

Key Insights and Ideas

  • The Power of Merging: A significant insight is that merging similar rhetorical roles into broader categories before initial classification dramatically improves performance. This simplification allows the HSLN to learn more robust patterns.
  • Two-Stage Approach is Crucial: The combination of a sequential sentence classifier (HSLN) for broad categorization and a few-shot learner (SetFit) for fine-tuning specific distinctions proves highly effective.
  • Legal Text is Different: The paper reiterates that legal documents have unique characteristics (length, jargon, structure) that necessitate specialized NLP approaches, as standard models trained on general text may not perform optimally.
  • Text Cleaning is Not Always Beneficial: Surprisingly, the authors found that extensive text cleaning (removing HTML tags, punctuation, etc.) did not lead to significant improvements in F1 score, suggesting that the models can handle some level of noise or that the cleaning process inadvertently removed useful signals.

Key Results

The proposed method achieved a top F1 score of 0.8301 on the SemEval-2023 Task 6, surpassing the baseline score of 0.79 established by the task organizers. This represents a significant improvement and secured them the 6th rank in the competition. The classification report for the training data shows high F1 scores for classes like 'PREAMBLE' (0.9931) and 'NONE' (0.933), indicating strong performance on these prevalent roles. The authors noted challenges in differentiating between 'ARG_PETITIONER' and 'ARG_RESPONDENT' due to limited training data and class similarity.

Limitations and Future Work

The authors acknowledge that differentiating between closely related roles like 'ARG_PETITIONER' and 'ARG_RESPONDENT' remains a challenge, likely due to insufficient training data for these specific classes. They also note that text cleaning provided minimal improvements.

Future work could focus on:

  • Developing more sophisticated methods for handling class imbalance and similarity.
  • Exploring other transformer-based models or architectures for improved performance.
  • Investigating the impact of different text cleaning strategies or feature engineering.
  • Extending this approach to other legal NLP tasks such as summarization, judgment prediction, and legal entity recognition, particularly in developing countries facing increasing legal backlogs.

Summary updated

BibTeX
@inproceedings{kataria2023nlp,
  author = {Harsh Kataria and Ambuje Gupta},
  title = {{NLP-Titan at SemEval-2023 Task 6: Identification of Rhetorical Roles Using Sequential Sentence Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure