Exploring Graph Neural Networks for Indian Legal Judgment Prediction

Mann Khatri, Mirza Yusuf, Yaman Kumar, Rajiv Ratn Shah, Ponnurangam KumaraguruarXiv 2023 · DOI · arXiv

Abstract

This paper presents a Graph Neural Network (GNN) framework designed to address the judicial backlog in India by predicting case outcomes and legal citations. By modeling the legal landscape as a heterogeneous graph of cases, acts, and temporal data, the researchers demonstrate that structural relationships significantly enhance prediction accuracy. The best-performing model, utilizing legally pretrained XLNet embeddings, achieved a macro F1 score of 75%, highlighting the importance of relational context in legal document analysis.

Introduction

The Indian judiciary is currently grappling with a staggering backlog of pending cases, exacerbated by a low judge-to-case ratio. To help expedite the legal process, researchers are investigating Legal Judgment Prediction (LJP)—automated systems that suggest case outcomes based on factual evidence and historical precedents. While previous attempts at LJP have treated legal documents as isolated pieces of text, the law is inherently relational. A single judgment often relies on a network of prior citations and specific constitutional acts. This study addresses the challenge of capturing this "intrinsic graph structure" to improve prediction accuracy and assess the fairness of automated legal decisions.

Methodology

The authors employed a GraphSAGE architecture, a popular GNN framework, to treat LJP as a binary node classification problem (predicting whether a case is "accepted" or "rejected").

Graph Construction and Edge Variations

The graph was constructed with heterogeneous nodes, including cases and "characteristic nodes" for specific Judicial Acts and Time (years). The researchers experimented with different citation directions to optimize message passing: Directed (case A cites B), Reverse-Directed (message passing from cited to citer), and Undirected.

Embedding Strategies

Four types of node features were compared to determine the most effective representation of legal text:

  1. Random initializations
  2. Vanilla XLNet
  3. XLNet pretrained on legal text
  4. A hierarchical embedding approach

Datasets

The study primarily utilized the Indian Legal Documents Corpus (ILDC), containing 35,000 Indian Supreme Court cases. To create a robust citation network, the authors used the IndianKanoon API to extract and add 24,907 additional cited cases, enabling the GNN to navigate a more complete relational structure.

Results and Discussion

Impact of Pretraining and Graph Structure

The most significant performance boost came from using XLNet embeddings specifically pretrained on legal judgment tasks. This suggests that "legal-speak" has a unique semantic structure that general-purpose language models struggle to capture without fine-tuning. As shown in the results table, the pretrained embeddings consistently outperformed other methods across various graph configurations.

Macro F1 scores for different embedding types and graph edge configurations.

Interestingly, while undirected edges proved efficient for predicting connections, adding specific "Time Nodes" to the graph had a slightly negative impact on performance. The authors conclude that while time is a factor in law, its explicit inclusion as a node type introduced more complexity than clarity for the model.

To simulate real-world utility, the researchers tested "Forward" training (training on past cases to predict future ones). In these scenarios, the F1 score steadily increased as more training data (years) were added, confirming that the model benefits from a larger historical context. The model also demonstrated high proficiency in Link Prediction (predicting which case will cite another), achieving an ROC score of over 80%.

Performance scores across different time intervals for various model configurations.

Fairness Analysis

A key component of the study was a redaction analysis to evaluate potential bias. By removing gendered pronouns and names, the researchers found that the model's output did not change significantly. This suggests the GNN focuses on legal merits and factual evidence rather than demographic "persona" data, an encouraging sign for the fairness of automated legal tools.

Conclusion and Future Work

The proposed GNN framework achieves a 75% macro F1 score for judgment prediction by leveraging the relational nature of the law. However, the authors acknowledge that the model currently operates as a "black box." Future work will focus on improving explainability, expanding the dataset to lower courts where backlogs are most severe, and developing multi-task learning models to simultaneously predict judgments, relevant acts, and potential penalties.

Summary updated

Abstract

The burdensome impact of a skewed judges-to-cases ratio on the judicial system manifests in an overwhelming backlog of pending cases alongside an ongoing influx of new ones. To tackle this issue and expedite the judicial process, the proposition of an automated system capable of suggesting case outcomes based on factual evidence and precedent from past cases gains significance. This research paper centres on developing a graph neural network-based model to address the Legal Judgment Prediction (LJP) problem, recognizing the intrinsic graph structure of judicial cases and making it a binary node classification problem. We explored various embeddings as model features, while nodes such as time nodes and judicial acts were added and pruned to evaluate the model's performance. The study is done while considering the ethical dimension of fairness in these predictions, considering gender and name biases. A link prediction task is also conducted to assess the model's proficiency in anticipating connections between two specified nodes. By harnessing the capabilities of graph neural networks and incorporating fairness analyses, this research aims to contribute insights towards streamlining the adjudication process, enhancing judicial efficiency, and fostering a more equitable legal landscape, ultimately alleviating the strain imposed by mounting case backlogs. Our best-performing model with XLNet pre-trained embeddings as its features gives the macro F1 score of 75% for the LJP task. For link prediction, the same set of features is the best performing giving ROC of more than 80%
BibTeX
@misc{khatri2023exploring,
  author = {Mann Khatri and Mirza Yusuf and Yaman Kumar and Rajiv Ratn Shah and Ponnurangam Kumaraguru},
  title = {{Exploring Graph Neural Networks for Indian Legal Judgment Prediction}},
  year = {2023},
  eprint = {2310.12800},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.12800},
}
Figure