TL;DR
This paper presents a novel approach for classifying sentences in legal documents into rhetorical roles, crucial for downstream legal NLP tasks. By combining RoBERTa's powerful sentence embeddings with Hierarchical Dirichlet Process (HDP) topic modeling, the system achieves a strong performance, ranking 12th in the SemEval-2023 Task 6. This work highlights the benefit of integrating topical information with contextual embeddings for improved legal text understanding.
What Problem Does This Paper Address?
The legal domain is characterized by lengthy, unstructured documents filled with specialized jargon. Effectively processing these documents for tasks like summarization, outcome prediction, or precedent search requires understanding the function of each sentence within the document's overall structure. This paper tackles the challenge of Rhetorical Role Prediction, which involves assigning a specific label (e.g., "Facts," "Issues," "Ruling") to each sentence in a legal judgment. This is a form of sequential sentence classification, where the order and context of sentences are critical, making it more complex than standard text classification. A key challenge in this domain is the limited availability of annotated corpora, further complicating the development of robust NLP models.
Methods
The proposed system, LRL_NC, employs a hierarchical sequence encoding approach. The core innovation lies in augmenting pre-trained transformer embeddings with topic modeling features.
- Sentence Representation: The system starts by using a fine-tuned RoBERTa model to generate initial sentence embeddings. This captures rich contextual information within each sentence.
- Topic Modeling Integration: To incorporate broader topical context from the entire corpus, a Hierarchical Dirichlet Process (HDP) topic model is trained on the training documents. HDP learns the number of topics automatically and generates a K-dimensional vector for each sentence, representing its topical composition.
- Combined Features: The RoBERTa embeddings and HDP topic proportions are concatenated to create a richer, combined sentence representation.
- Contextualization: A BiLSTM layer is then applied to these combined sentence representations to capture sequential dependencies and generate contextualized sentence representations.
- Final Classification: A Conditional Random Field (CRF) layer is used as the final classifier. CRF is well-suited for sequence labeling tasks as it models dependencies between adjacent labels, leading to more coherent predictions.
The authors also explored variations, including discarding HDP features or using a parametric topic model (CTM) instead of HDP.
Datasets
The primary dataset used is the ILDC (Indian Legal Documents Corpus), which was introduced by Malik et al. (2021b). This corpus contains legal documents annotated with 13 different Rhetorical Role labels, as detailed in Table 1 of the paper. The dataset was used for both training and validation, with experiments conducted on both validation and test subsets.
Key Insights and Ideas
- Topic Modeling Enhances Contextual Embeddings: The most significant insight is that augmenting powerful pre-trained embeddings like RoBERTa with topic modeling features (HDP) leads to improved performance in legal document sentence classification. This suggests that while transformers capture local context well, explicit topical information helps the model understand the broader thematic structure of legal texts.
- HDP Outperforms CTM: The paper found that the non-parametric HDP model, which automatically determines the number of topics, was more effective than the parametric Correlated Topic Model (CTM) for extracting additional features in this task.
- CRF is Crucial for Sequential Labeling: The use of a CRF layer at the end of the pipeline is essential for effectively modeling label dependencies and achieving state-of-the-art results in sequential sentence classification tasks like Rhetorical Role Prediction.
Key Results
The proposed system, RoBERTa + HDP + BiLSTM + CRF, achieved a micro-F1 score of 0.7980 on the test subset of the ILDC dataset. This performance secured the team the 12th rank in the official SemEval-2023 Task 6 leaderboard.
Key comparative results from the paper:
- RoBERTa + HDP + BiLSTM + CRF: 0.7980 (Test Set)
- RoBERTa + BiLSTM + CRF (without HDP): 0.7809 (Test Set)
- RoBERTa + CTM + BiLSTM + CRF: 0.7593 (Test Set)
These results clearly demonstrate the benefit of incorporating HDP features.
Limitations and Future Work
- Limitations: The paper acknowledges that while their system performs well, it was not the top-ranked system. The authors also mention that some variations of their scheme involved RoBERTa models that were not fine-tuned for the MLM task on the ILDC dataset, which might have impacted performance.
- Future Work: The authors suggest exploring other methods to incorporate corpus information gained from topic modeling to further improve sequential sentence classification. They also propose investigating alternative ways to leverage topic modeling beyond simply concatenating features.
Summary updated