TL;DR
This paper introduces a novel multi-level encoder architecture for predicting legal case judgments and generating explanations. By combining BERT with transformer encoder layers and RNNs, the system achieved impressive results in the SemEval-2023 LegalEval challenge, ranking second for judgment prediction and seventh for explanation generation.
What Problem Does This Paper Address?
The legal system faces a growing backlog of cases, making it challenging for judges to process them efficiently. This paper addresses the need for automated systems that can assist legal professionals by predicting the outcome of a case and providing a clear, understandable explanation for that prediction. This is particularly important for individuals unfamiliar with legal jargon and proceedings, enabling them to make informed decisions. The specific challenge tackled is within the SemEval-2023 Task 6: LegalEval, focusing on Indian legal cases from the Supreme Court of India. The task is split into two sub-tasks: predicting the judgment (Sub-task C-1) and providing an explanation for that prediction (Sub-task C-2).
Methods
The core of the proposed solution is a three-level encoder-based classification architecture.
- Level 1: Custom Fine-tuning: The system starts by fine-tuning a pre-trained BERT model. To handle the long legal documents (averaging 20,000 tokens), documents are split into chunks of 510 tokens with overlaps. These chunks are then passed through the BERT encoder.
- Level 2: Extracting Chunk Embeddings: Embeddings from the last four layers of the fine-tuned BERT encoder are extracted for each chunk. These chunk embeddings are then aggregated to form a representation of the entire document.
- Level 3: Post-processing Chunk Embeddings: To capture inter-chunk relationships, transformer encoder layers are applied. This is followed by max-pooling, a linear layer, and an RNN (specifically BiLSTM) layer. Finally, a linear layer with a sigmoid activation function performs the binary classification for judgment prediction.
For explanation generation (Sub-task C-2), the authors adapt the concept of "occlusion sensitivity." This involves systematically masking parts of the input (chunks, then sentences) and observing the impact on the model's prediction probability. Greater decreases in probability indicate higher importance, allowing the system to identify the most relevant sentences as explanations.
Datasets
The primary dataset used is from SemEval Task-6: LegalEval. This dataset consists of Indian legal cases from the Supreme Court of India, with original decisions removed. Each document is labeled as "Rejected" (0) or "Accepted" (1). The dataset is further divided into:
- Multi: Cases with multiple petitions and potentially different decisions. The final decision is used as the document label.
- Single: Cases with a single decision for all petitions.
The LegalEval dataset has training, validation, and test splits. Notably, for Sub-task C-2, there is no annotated explanation data, making it an extractive explanation task. The authors also experimented with the Indian Legal Document Corpus (ILDC), which was used for testing and to approximate performance on a real-world setting.
Key Insights and Ideas
- Hierarchical Processing for Long Documents: The chunking strategy is crucial for handling long legal documents that exceed the input limits of standard BERT models.
- Leveraging Multiple Encoder Layers: Extracting embeddings from the last four BERT layers, rather than just the last, provides richer representations for downstream processing.
- Combining Transformer and RNNs: The fusion of transformer encoder layers for inter-chunk attention and RNNs for sequential processing proves effective in capturing complex document structures.
- Occlusion Sensitivity for Explainability: Adapting occlusion sensitivity from computer vision to text allows for a model-agnostic approach to identifying important segments for explanation, without requiring explicit annotation.
- Dataset Skew Impact: Analysis revealed a significant skew in the distribution of case filing/appeal dates between the training and validation/test sets, potentially impacting model generalization.
Key Results
- Sub-task C-1 (Judgment Prediction): The system ranked 2nd out of 11 participating teams with a macro-F1 score of 0.7228.
- Sub-task C-2 (Explanation Generation): The system ranked 7th out of 11 participating teams with a ROUGE-2 score of 0.0428.
- Ablation studies showed that using embeddings from the last four BERT layers and adding RNNs improved performance.
- Oversampling the training data to balance class labels led to a significant improvement in m-F1 scores, reaching up to 75.25% on the validation set.
Limitations and Future Work
- Dataset Skew: The authors identified a significant skew in the temporal distribution of cases between training and test sets, hypothesizing that this lack of recent data hinders model prediction.
- Model Dependency of Explanation: The explanation extraction algorithm is model-dependent, meaning its effectiveness is tied to the classification model's ability to discern input importance.
- Future Work: The authors plan to develop more general transformer architectures for decision prediction and explanation, exploring both abstractive and extractive approaches for legal texts. They also aim to investigate further into the dataset skew and its impact.
Summary updated