CSECU-DSG at SemEval-2023 Task 6: Segmenting Legal Documents into Rhetorical Roles via Fine-tuned Transformer Architecture

Fareen Tasneem, Tashin Hossain, Jannatun Naim, Abu Nowshed ChyACL 2023

TL;DR

This paper tackles the challenge of automatically segmenting legal documents into semantically meaningful parts based on their rhetorical roles. The authors leverage a fine-tuned Legal-BERT model to classify sentences into 13 distinct rhetorical roles, achieving a respectable F1-score on a specialized legal corpus. This work contributes to making complex legal texts more accessible and processable by automated systems.

What Problem Does This Paper Address?

Legal documents are notoriously complex, varying greatly in structure and content across jurisdictions and domains. This variability makes it difficult for automated systems to perform tasks like summarization, analysis, and semantic search effectively. The authors aim to address this by segmenting legal documents into rhetorically meaningful units, where each segment serves a specific purpose within the document's overall argument. Identifying these rhetorical roles can significantly aid in extracting crucial information and improving the efficiency and accessibility of legal text processing.

Methods

The core of the proposed approach is the use of a fine-tuned Legal-BERT model. Legal-BERT, a variant of BERT pre-trained on a large corpus of Harvard Law legal documents, is specifically designed to understand the nuances of legal language. The authors treat the task as a multi-class sequence classification problem. Each legal document is first segmented into individual sentences. These sentences are then tokenized and fed into the fine-tuned Legal-BERT model, which predicts one of 13 predefined rhetorical role labels for each sentence. The model was trained using the AdamW optimizer with a learning rate of 1e-5 over 20 epochs.

Datasets

The study utilizes the Rhetorical Roles Corpus provided for SemEval-2023 Task 6, Subtask A. This corpus consists of 354 English Indian legal judgment documents, totaling 40,305 sentences. Each sentence is annotated with one of 13 rhetorical roles, including preamble, facts, analysis, and statute. The dataset was split into training, development, and testing sets, with a summary provided in Table 1.

Key Insights and Ideas

  • Legal-BERT's Specialization: The paper highlights the advantage of using a domain-specific pre-trained model like Legal-BERT for legal text classification tasks, suggesting it captures the critical context of legal sentences better than general-purpose models.
  • Rhetorical Roles as Building Blocks: The fundamental idea that segmenting documents by rhetorical roles can unlock more sophisticated downstream NLP tasks in the legal domain is reinforced.
  • Challenges of Imbalance: The authors acknowledge that dataset imbalance can significantly impact performance, leading to a considerable difference in prediction accuracy for certain rhetorical roles.

Key Results

The proposed system, CSECU-DSG, achieved a micro F1-score of 0.6465 on the test set of the judgment documents. While this performance is respectable and competitive with some participants, it falls short of the top-performing teams in the SemEval-2023 task. The paper also includes an error analysis, showing instances where the model correctly identified statutes but misclassified facts as preambles, and vice-versa, indicating the subtle distinctions the model struggles with. Figure 3 provides a statistical breakdown of correctly predicted labels versus gold labels on the development set, illustrating areas of strength and weakness.

Limitations and Future Work

The authors acknowledge that their model's performance is mediocre compared to the top-performing teams, indicating room for improvement. They also point out that the dataset's imbalance for certain rhetorical roles contributes to performance gaps.

For future work, the authors intend to:

  • Extend the corpus to include legal domains with different structures.
  • Address the other two subtasks of SemEval-2023 Task 6, suggesting a broader ambition to tackle multiple facets of legal text understanding.

Summary updated

BibTeX
@inproceedings{tasneem2023csecu,
  author = {Fareen Tasneem and Tashin Hossain and Jannatun Naim and Abu Nowshed Chy},
  title = {{CSECU-DSG at SemEval-2023 Task 6: Segmenting Legal Documents into Rhetorical Roles via Fine-tuned Transformer Architecture}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure