Steno AI at SemEval-2023 Task 6: Rhetorical Role Labelling of Legal Documents using Transformers and Graph Neural Networks

Anshika Gupta, Shaz Furniturewala, Vijay Kumari, Yashvardhan SharmaACL 2023

TL;DR

This paper tackles the challenge of automatically understanding legal documents by classifying sentences based on their rhetorical role. The authors propose a novel "Context-based LEGAL-BERT" model that significantly outperforms previous methods by incorporating sentence context, achieving a 71.02% accuracy on a challenging legal text dataset.

What Problem Does This Paper Address?

Legal documents, particularly court judgments, are notoriously dense, jargon-filled, and lengthy. Extracting meaningful information and understanding the structure of these documents is a labor-intensive task for legal professionals. Rhetorical Role Labelling (RRL) aims to automate this by classifying each sentence according to its semantic function within the document (e.g., identifying arguments, facts, or legal reasoning). This is crucial for downstream tasks like legal summarization, precedent search, and outcome prediction. However, existing RRL models, even those based on powerful transformer architectures like BERT, have struggled to achieve high accuracy on complex legal texts, indicating a need for more sophisticated approaches.

Methods

The researchers experimented with several approaches, all built upon the LEGAL-BERT family of models, which are pre-trained on a large corpus of legal texts.

  • Baseline Models: They started with a direct classification using LEGAL-BERT's default classifier and then explored using the extracted CLS tokens (a fixed embedding representing the sentence's semantic understanding) from LEGAL-BERT's final hidden layer. These CLS tokens were fed into multi-layer neural networks as a baseline.
  • Graph-Based Approaches: They modeled the relationships between sentences as a graph. Edges were formed between sentences if their CLS tokens had a cosine similarity above a certain threshold (0.5), with the similarity score acting as the edge weight. On this graph, they applied:
  • Label Diffusion: A random walk algorithm to propagate labels based on neighbor information.
  • Graph Convolutional Networks (GCN): Inspired by BERTGCN, they used LEGAL-BERT embeddings as node representations and applied a two-layer GCN to learn optimal label weights from one-hop neighbors. This approach was computationally expensive, limiting its application to a subset of the training data.
  • Context-Based LEGAL-BERT: This was their most promising approach. They preprocessed sentences by removing stopwords and then created a 5-sentence input for each target sentence (the sentence itself, its two preceding, and two succeeding sentences). These context-aware inputs were then fed into a fine-tuned LEGAL-BERT model with its default classifier.

Datasets

The primary dataset used in this work is derived from publicly available Indian Supreme Court Judgements. This dataset, previously introduced by Kalamkar et al. (2022), consists of:

  • 244 training documents
  • 30 validation documents
  • 50 test documents
  • A total of 36,023 sentences, categorized into 13 semantic classes.

Key Insights and Ideas

  • Context is King: The most significant finding is that incorporating the surrounding sentences (context) dramatically improves the accuracy of rhetorical role labelling. This suggests that a sentence's function is often dependent on its neighbors, not just its own content.
  • LEGAL-BERT's Strength: The LEGAL-BERT model, pre-trained on legal texts, provides a strong foundation for understanding legal language. However, its raw output needs further refinement to capture nuanced sentence roles.
  • Graph Limitations: While graph-based approaches are intuitive for capturing sentence relationships, they proved computationally intensive and did not yield significant improvements over simpler methods in this study, possibly due to limitations in sentence representation or computational constraints.
  • Efficiency Matters: The Context-based LEGAL-BERT model not only achieved the highest accuracy but was also less resource-intensive than the graph-based methods, making it a more practical solution.

Key Results

The Context-based LEGAL-BERT model achieved the highest accuracy on the validation dataset at 71.02%. This represents a significant improvement of 5.5 percentage points over the baseline LEGAL-BERT model. Other models performed as follows:

  • LEGAL-BERT: 65.56%
  • LEGAL-BERT + Classifier: 67.15%
  • Graph Label Diffusion: 66.34%
  • GCN: 67.42%

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Computational Constraints: The graph-based approaches, particularly the GCN, were computationally expensive, preventing them from being trained on the entire dataset. This might have limited their potential performance.
  • Dataset Size: While the dataset is substantial, larger datasets could further improve model performance.
  • Future Work:
    • Applying GCNs to the complete dataset.
  • Exploring other classification methods (custom neural networks, label diffusion) on context-based CLS tokens.
  • Investigating models capable of handling longer contexts (e.g., Longformer) to incorporate even more surrounding sentences.

Summary updated

BibTeX
@inproceedings{gupta2023steno,
  author = {Anshika Gupta and Shaz Furniturewala and Vijay Kumari and Yashvardhan Sharma},
  title = {{Steno AI at SemEval-2023 Task 6: Rhetorical Role Labelling of Legal Documents using Transformers and Graph Neural Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure