TL;DR
This paper presents a system for predicting rhetorical roles in legal judgments, achieving a 73.35 F1 score. The core innovation lies in combining local sentence-level predictions from language models with global document-level features, particularly the temporal distribution of roles within a judgment.
What Problem Does This Paper Address?
Legal documents, like court judgments, are complex and lengthy. Understanding their structure is crucial for legal professionals and AI systems alike. This paper tackles SemEval-2023 Task 6, which focuses on predicting "Rhetorical Roles" (RRs) for pre-segmented paragraphs within legal judgments. These RRs categorize parts of a judgment (e.g., PREAMBLE for introductory statements, RATIO for the court's reasoning) to provide a structured overview. The challenge lies in effectively integrating information from individual paragraphs with the overall document context, especially given the long nature of legal texts which can strain standard language models.
Methods
The authors propose a multi-faceted approach that combines several models:
- Language Model (LM) Scores: They fine-tune a RoBERTa-base model to predict RRs for individual sentences. For paragraphs containing multiple sentences, probabilities are multiplied.
- Position Scores: This is a key contribution. They model the likelihood of a specific RR appearing at different relative positions within a judgment. This is achieved by fitting a mixture of Beta distributions to the observed positions of RRs in the training data. This explicitly captures the temporal structure of legal documents.
- Global Transition Scores: This model learns conditional probabilities of RRs appearing given the set of RRs already predicted in the document. It uses a Multi-Layer Perceptron (MLP) to generalize beyond observed transitions, aiming to capture structural constraints.
- Local Transition Scores: This simpler model captures whether specific transitions between consecutive RRs (e.g., from RR A to RR B) are observed in the training data.
- Domain Knowledge: Hand-crafted lexical rules are incorporated to enforce or block certain RR predictions based on specific keywords or phrases found in the text.
During inference, these scores are combined using a Viterbi algorithm to find the most probable sequence of RRs for the entire judgment.
Datasets
The primary dataset used is the SemEval-2023 Task 6: LegalEval dataset. This dataset consists of:
- 245 segmented and labeled Indian judgments (English) for training.
- 30 segmented and labeled judgments for development.
- 50 segmented but unlabeled judgments for testing.
The paper notes that test set labels are not released by the organizers, necessitating the use of the development set for ablation studies.
Key Insights and Ideas
- Temporal Distribution is King: The most significant insight is the crucial role of modeling the temporal distribution of RRs within a document (Position Scores). This suggests that understanding where a particular type of legal argument typically appears is as important as understanding the argument itself.
- Global Structure Matters: Beyond local sentence meaning, global document structure (captured by Position Scores and Global Transition Scores) significantly boosts performance.
- LM is Foundational, but Not Sufficient: While fine-tuned LMs are essential for extracting semantic meaning, they alone are not enough. Integrating other structural and domain-specific information is vital for optimal performance.
- Local Transitions Less Impactful: The authors found that simple local transition models (predicting if consecutive sentences share an RR) were less effective, likely due to data imbalance and the limited expressiveness of their chosen model.
- Domain Knowledge Adds Value: Even with sophisticated models, hand-coded lexical rules derived from legal expertise provide a small but consistent performance improvement.
Key Results
- Overall Performance: The submitted system achieved a 73.35 micro-averaged F1 score on the test set, ranking 17th out of 27 participants.
- Ablation Study:
- The LM alone achieved 68.01 F1.
- Adding Position Scores boosted performance to 75.30 F1.
- Including Global Transition Scores further improved it to 76.52 F1.
- Local Transition Scores provided minimal additional gain.
- Domain Knowledge added a small improvement, bringing the final score to 77.08 F1 (using roberta-base).
- Model Contributions: The average contribution of each model (when added to other models) was: Position Scores (+2.98 F1), Global Transition Scores (+2.06 F1), Domain Knowledge (+0.64 F1), and Local Transition Scores (+0.25 F1).
Limitations and Future Work
- Data Sparsity for Global Transitions: The authors acknowledge that estimating parameters for all possible global transition distributions is challenging due to data sparsity. Their MLP approach helps generalize, but more expressive models might be beneficial.
- Local Transition Model Limitations: The current local transition model is deemed not expressive enough. Future work could explore more sophisticated models for local transitions.
- LM Limitations: The paper suggests that showing LMs more context could improve their performance. Exploring different LM setups or fine-tuning hyperparameters might also be necessary.
- Future Directions: The authors propose improving LM performance by providing more context and developing better models for local transitions. They also plan to make their code publicly available.
Summary updated