TL;DR
This paper tackles the challenge of automatically structuring legal documents by predicting rhetorical roles (like "preamble" or "fact") for each sentence. The authors propose a novel hierarchical model, LEGAL-BERT-based Hierarchical BiLSTM with CRF (HLBERT-CRF), which leverages a legal-domain specific pre-trained language model and a multi-level encoding approach to achieve strong performance, ranking 7th out of 27 teams in the SemEval-2023 Task 6.
What Problem Does This Paper Address?
Legal documents are complex and lengthy, making it difficult for legal professionals to efficiently extract crucial information. To address this, SemEval-2023 Task 6 introduced LegalEval, focusing on understanding legal texts. One key subtask, Rhetorical Roles (RR) prediction, aims to automatically assign semantic labels (e.g., preamble, fact, ratio) to segments of legal text, effectively structuring unstructured documents. This is a fundamental step for downstream AI applications like legal summarization and precedent search. The challenge lies in the specialized vocabulary and the sheer length of legal texts, which often exceed the input limits of general-purpose pre-trained language models.
Methods
The proposed method, HLBERT-CRF, employs a hierarchical architecture to process legal documents. It consists of two main encoders:
- Word-level Encoder: This component uses LEGAL-BERT, a pre-trained language model specifically trained on legal corpora, to generate initial word embeddings. These embeddings are then fed into a bidirectional Long Short-Term Memory (BiLSTM) network to capture sentence-level representations.
- Sentence-level Encoder: The sentence representations from the word-level encoder are further processed. An attentive pooling mechanism is used to create a unified sentence embedding. This is followed by another BiLSTM layer to model the relationships between sentences within a document, resulting in a document representation.
- CRF Classifier: Finally, a Conditional Random Field (CRF) layer is used to predict the rhetorical role label for each sentence, considering the dependencies between labels.
Additionally, the authors implemented a hard-voting ensemble strategy, where multiple HLBERT-CRF models trained with different random seeds are combined to make the final prediction, aiming to improve robustness and performance.
Datasets
The experiments were conducted on a corpus collected from Indian Court websites, including Supreme Court, High Courts, and district court judgments. This corpus was split into training, development, and test sets. The dataset contains document IDs, annotations (sentence ID, text, position), and the actual judgment text. The task involves classifying sentences into 13 rhetorical role labels, such as preamble, fact, and ratio.
Key Insights and Ideas
- Legal-Specific Pre-training is Crucial: Using LEGAL-BERT, a model pre-trained on legal data, significantly outperforms general-purpose pre-trained models (like BERT, RoBERTa, XLNet, ALBERT) when integrated into the HLBERT-CRF framework. This highlights the importance of domain-specific knowledge for legal NLP tasks.
- Hierarchical Structure Matters: The hierarchical BiLSTM architecture, processing information at both word and sentence levels, is essential for effectively modeling legal documents. Ablation studies showed that removing either the word-level or sentence-level BiLSTM significantly degraded performance.
- BiLSTM Outperforms Transformers for this Task: Replacing BiLSTM layers with Transformer layers in the hierarchical structure led to a decrease in performance, suggesting that BiLSTM's sequential modeling is more suitable for capturing the hierarchical nature of legal text in this context.
- Hard Voting Improves Performance: The ensemble strategy of hard voting across multiple models provided a slight but noticeable improvement in performance, indicating its utility for boosting accuracy.
Key Results
The proposed HLBERT-CRF model achieved a weighted micro-F₁ score of 0.8146 on the test set. This performance secured the team 7th rank out of 27 participants in SemEval-2023 Task 6 (Subtask A). The hard-voting ensemble strategy slightly outperformed the single model, achieving 0.8146 compared to 0.8140.
Limitations and Future Work
The authors acknowledge that the primary focus was on rhetorical role prediction. For future work, they plan to explore more complex legal text-based NLP tasks and investigate more efficient structures and explainable modules. The paper also implicitly suggests that further research into adapting pre-trained models for the specific nuances of legal language could yield even better results.
Summary updated