Automatic Linking of Judgements to UK Supreme Court Hearings

Hadeel Saadany, Constantin OrasanEMNLP 2023

TL;DR

This paper introduces J-HAL, an AI system that automatically links segments of UK Supreme Court judgments to corresponding timestamps in video recordings of the hearings. This innovation aims to improve access to justice by allowing legal professionals and the public to quickly navigate lengthy court proceedings and pinpoint crucial arguments that shaped the final decisions.

What Problem Does This Paper Address?

The UK Supreme Court, as the final court of appeal, produces landmark legal decisions that significantly impact UK law. These decisions are accompanied by extensive video recordings of the hearings, which are invaluable for understanding the judicial process, legal training, and policy development. However, the sheer length of these recordings (often spanning hours over multiple days) makes it incredibly time-consuming and difficult for legal professionals to extract key information. Current methods rely on human transcription, which is slow and expensive. This paper addresses the challenge of efficiently bridging the gap between the written judgment and the spoken word in court hearings, making this rich legal material more accessible and navigable.

Methods

The J-HAL system employs a two-stage pipeline. The first stage, which is beyond the scope of this paper, focuses on improving the quality of speech-to-text transcriptions using custom language models. The core contribution of this paper lies in the second stage: an information retrieval (IR) system designed to automatically link paragraphs from a judgment text to relevant time segments within the transcribed video.

The authors frame this as a document-to-document retrieval task. They treat each paragraph of the judgment as a "query" and the entire transcript of the court hearing as the "corpus." A key challenge is the difference in language registers between formal written judgments and informal spoken transcripts.

For the IR system, they explore both traditional keyword-based methods (Okapi BM25) and more advanced embedding-based methods. Notably, they experiment with various pre-trained language models, including GloVe, Entailment Search (MiniLM), Legal BERT, MS MARCO (for asymmetric search), and crucially, OpenAI's GPT text-embedding-ada-002.

A significant aspect of their methodology is the development of a zero-shot IR approach for creating an evaluation dataset. Due to the prohibitive cost of human annotation for every possible link, they use IR models to generate candidate links and then have human annotators evaluate a subset. They also detail a model optimization step where they fine-tune the GPT embeddings on their legal dataset to improve domain specificity.

Datasets

The authors compiled a dataset consisting of:

  • 7 case judgments: Approximately 1.4 million tokens scraped from the UK Supreme Court website.
  • 53 hours of video material: Obtained from the UK National Archive for the selected cases.

This data was used to train their custom speech-to-text model and subsequently for evaluating the IR linking models. They also created a manually annotated dataset of 3620 judgement-to-transcript links for evaluation.

Key Insights and Ideas

  • Bridging the Spoken-Written Divide: The paper highlights the linguistic challenges of linking formal written legal text with informal spoken transcripts, noting differences in lexical density and the presence of meta-linguistic elements in speech.
  • GPT Embeddings Shine: OpenAI's GPT text-embedding-ada-002 model, particularly after customization, demonstrated superior performance in linking judgement segments to relevant video timespans compared to other tested models.
  • Zero-Shot for Scalability: The adoption of a zero-shot IR approach for dataset creation is a pragmatic solution to the immense annotation burden in this domain, enabling scalable evaluation.
  • Customization is Key: Fine-tuning pre-trained embeddings (like GPT) on domain-specific data significantly improves linking accuracy, suggesting that general-purpose models can be effectively adapted for specialized legal tasks.
  • Recall Over Precision: For this application, recall (finding as many relevant links as possible) is more critical than precision (ensuring all found links are relevant), as the goal is to provide a comprehensive set of bookmarks for users.

Key Results

  • The GPT model, after customization, achieved the highest performance, improving the overlap between cosine similarity distributions for relevant and irrelevant links from 70.5% to 73.0%.
  • In terms of Mean Average Precision (MAP), the GPT model consistently outperformed other models across various hit counts (MAP@5, MAP@10, MAP@15) on both a single-case dataset and the entire dataset. For instance, on the entire dataset, GPT achieved a MAP@15 of 0.914, significantly higher than BM25 (0.902) and other embedding models.
  • The paper's UI, powered by this linking system, has shown interest from legal academics and professionals for its utility as a bookmarking tool.

Limitations and Future Work

  • System Focus: The paper focuses on the second stage of the pipeline (linking), with the speech-to-text transcription stage being described in prior work.
  • No Direct Answers: The system provides navigation and bookmarking, but it does not explicitly answer legal questions or provide legal advice.
  • Future Directions: The authors plan to expand their annotated linking dataset and explore linking based on common legal entities like articles, legal provisions, and similar cases. They also aim to adapt this methodology for other domains, such as lecture recordings or business meetings.

Summary updated

BibTeX
@inproceedings{saadany2023automatic,
  author = {Hadeel Saadany and Constantin Orasan},
  title = {{Automatic Linking of Judgements to UK Supreme Court Hearings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure