Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courts

Shubham Kumar Nigam, Anurag Sharma, Danush Khanna, Noel Shallum, Kripabandhu Ghosh, Arnab Bhattacharya

Introduction

The Indian judiciary is currently grappling with a massive backlog of millions of cases. While Artificial Intelligence (AI) offers a potential solution through Legal Judgment Prediction (LJP), existing systems are often limited by the "black box" problem—providing verdicts without the legal reasoning necessary for practitioners—and a severe lack of high-quality, expert-annotated data specific to the Indian legal context.

This research introduces PredEx, a landmark dataset and framework designed to move beyond simple classification toward generating human-readable legal rationales. By applying instruction tuning to Large Language Models (LLMs), the study provides a path for AI to assist in navigating complex judicial proceedings with transparency.

The PredEx Dataset

At the heart of this work is the PredEx dataset, which comprises 15,222 legal documents sourced from the Supreme Court of India and various High Courts. Over an 18-month period, a team of 10 legal experts meticulously annotated these cases to include both predictions and detailed explanations.

Comparison of PredEx with other global legal datasets, highlighting its scale and expert-annotated explanation component.

As shown in the comparison above, PredEx significantly expands upon previous efforts like ILDC, which contained only 56 explanation annotations. With over 15,000 such annotations, PredEx is now the largest dataset of its kind for the Indian legal domain, providing a robust benchmark for future research.

Methodology

The researchers employed a multi-pronged approach to bridge the gap between raw legal text and structured reasoning:

  • Baseline Transformers: Traditional encoder models such as InLegalBERT and RoBERTa-Large were tested using a "moving window" chunking strategy to handle the extreme length of legal documents.
  • Instruction Tuning: The core innovation involved fine-tuning LLMs like Llama-2-7B and Gemini Pro using 16 distinct instruction sets validated by legal experts. This trained the models to perform both extractive tasks (identifying pivotal sentences) and abstractive tasks (summarizing legal logic).

Key Results and Insights

The study revealed a fascinating dichotomy between prediction accuracy and explanatory depth. While generative LLMs are superior at reasoning, traditional discriminative models still lead in pure classification accuracy.

Performance comparison of LM-based and LLM-based models on prediction and explanation tasks.

As detailed in the results table, RoBERTa-Large achieved the highest prediction performance with a Macro F1 score of 0.7827. However, the instruction-tuned Llama-2-7B model demonstrated significant capabilities in generating explanations, achieving a BERTScore of 0.6909 and receiving expert ratings (2.84 to 3.30 on a 5-point scale) that suggest they are approaching human-standard rationales.

Limitations and Future Directions

Despite these advances, challenges remain. Many Indian legal documents exceed the token limits of current LLMs, and generative models still occasionally suffer from hallucinations or inconsistencies between their reasoning and final verdict.

Future work will focus on developing a domain-specific LLM for the Indian legal system and implementing Reinforcement Learning from Human Feedback (RLHF) to better align AI-generated explanations with the specific stylistic and logical requirements of the Indian Bar.

Summary updated

Abstract

In the era of Large Language Models (LLMs), predicting judicial outcomes poses significant challenges due to the complexity of legal proceedings and the scarcity of expert-annotated datasets. Addressing this, we introduce \textbf{Pred}iction with \textbf{Ex}planation (\texttt{PredEx}), the largest expert-annotated dataset for legal judgment prediction and explanation in the Indian context, featuring over 15,000 annotations. This groundbreaking corpus significantly enhances the training and evaluation of AI models in legal analysis, with innovations including the application of instruction tuning to LLMs. This method has markedly improved the predictive accuracy and explanatory depth of these models for legal judgments. We employed various transformer-based models, tailored for both general and Indian legal contexts. Through rigorous lexical, semantic, and expert assessments, our models effectively leverage \texttt{PredEx} to provide precise predictions and meaningful explanations, establishing it as a valuable benchmark for both the legal profession and the NLP community.
BibTeX
@inproceedings{nigam2024legal,
  author = {Shubham Kumar Nigam and Anurag Sharma and Danush Khanna and Noel Shallum and Kripabandhu Ghosh and Arnab Bhattacharya},
  title = {{Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.255},
}
Figure