IRIT_IRIS_A at SemEval-2023 Task 6: Legal Rhetorical Role Labeling Supported by Dynamic-Filled Contextualized Sentence Chunks

Alexandre Gomes de Lima, Jose G. Moreno, Eduardo H. da S. AranhaACL 2023

TL;DR

This paper introduces a novel method for encoding legal sentences into numerical representations, called Dynamic-Filled Contextualized Sentence Chunks (DFCSC). By intelligently chunking sentences and their surrounding context, their approach significantly improves the performance of Transformer models on the Legal Rhetorical Role Labeling task, outperforming existing methods.

What Problem Does This Paper Address?

Understanding legal documents is a complex task due to their length and intricate structure. The Legal Rhetorical Role Labeling (Legal RRL) task aims to assign semantic roles (like facts, arguments, rulings) to individual sentences within these documents. This is crucial for tasks like document interpretation, summarization, and predicting court judgments. While pre-trained Transformer models have shown promise, they often struggle with long legal texts. Existing methods either process sentences in isolation, losing valuable inter-sentence context, or use chunking strategies that lead to truncated sentences and excessive padding, both of which degrade the quality of sentence embeddings and, consequently, model performance.

Methods

The core innovation is the Dynamic-Filled Contextualized Sentence Chunks (DFCSC) approach. Unlike previous chunking methods that either don't share sentences between chunks or use fixed numbers of core/edge sentences, DFCSC aims to create chunks that:

  • Avoid padding tokens: By dynamically filling chunk edges with contextual tokens from neighboring sentences, DFCSC minimizes the need for padding, which can dilute sentence representations.
  • Prevent truncation of core sentences: The primary goal is to include the entire core sentence, avoiding data loss.
  • Maximize context: DFCSC strategically includes surrounding sentences (edge sentences) to provide richer context for the core sentence being classified.

The assembly logic prioritizes including as many consecutive sentences as possible while ensuring a minimum number of shared tokens between adjacent chunks. This dynamic filling, based on token counts rather than sentence counts, allows for more flexible and context-rich chunk creation. The authors then evaluate two variants: DFCSC-SEP (using the separator token's hidden state for representation) and DFCSC-CLS (using the concatenation of hidden states of the special token marking the beginning of the chunk).

Datasets

The study uses the dataset from SemEval-2023 Task 6 (LegalEval), which comprises judgment documents from the Indian legal system. This dataset contains sentences manually labeled with one of 13 pre-defined rhetorical roles. The dataset is noted to be highly unbalanced and exhibits significant variation in sentence length.

Key Insights and Ideas

  • Context is King (for Legal RRL): The results strongly suggest that leveraging inter-sentence context is significantly more beneficial than processing sentences in isolation for Legal RRL.
  • Padding is the Enemy: The paper highlights how excessive padding tokens, a common issue in previous chunking methods, negatively impact sentence embedding quality and model performance. DFCSC's avoidance of padding is a key differentiator.
  • Dynamic Chunking Outperforms Fixed: The DFCSC approach, with its flexible chunk assembly, proves more effective than fixed-size or non-overlapping chunking strategies.
  • Longer Context, Better Embeddings: Models that can handle longer sequences (like Longformer) and benefit from richer contextual chunks (DFCSC) achieve the highest performance.
  • DFCSC-CLS vs. DFCSC-SEP: The DFCSC-CLS approach, which uses the beginning-of-chunk token representation, generally shows slightly better performance and higher training stability.

Key Results

  • DFCSC-based models consistently outperform baseline methods.
  • The best DFCSC model achieved a weighted F1 score of 0.8139, significantly outperforming the best baseline of 0.7573.
  • DFCSC models achieved improvements of 3.76% to 8.71% over strong baselines.
  • Models based on RoBERTa and Longformer benefited more from DFCSC than BERT-based models, suggesting their better ability to leverage contextual information.
  • DFCSC-SEP Longformer and DFCSC-CLS Longformer achieved the two highest scores among all tested models.
  • The proposed approach achieved 9th place in the overall SemEval-2023 Task 6 leaderboard.

Limitations and Future Work

  • Hyperparameter Tuning: While DFCSC simplifies hyperparameter tuning by requiring fewer parameters than some other chunking methods, the paper notes that the impact of the m (minimum edge tokens) hyperparameter is less pronounced than initially expected, suggesting that the absolute amount of context might be more important than its proportion.
  • Dataset Specificity: The current findings are based on a specific dataset (Indian legal judgments).
  • Future Work: The authors plan to apply DFCSC to other legal RRL datasets to further validate their findings and explore its applicability to a broader range of legal text understanding tasks.

Summary updated

BibTeX
@inproceedings{lima2023irit,
  author = {Alexandre Gomes de Lima and Jose G. Moreno and Eduardo H. da S. Aranha},
  title = {{IRIT\_IRIS\_A at SemEval-2023 Task 6: Legal Rhetorical Role Labeling Supported by Dynamic-Filled Contextualized Sentence Chunks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure