CDD: A Large Scale Dataset for Legal Intelligence Research

Changzhen Ji, Yating Zhang, Adam Jatowt, Haipang WuEMNLP 2023

TL;DR

This paper introduces the Court Debate Dataset (CDD), a novel, large-scale dataset of real-world judicial dialogues designed to advance Legal Intelligence research. By providing rich annotations for features, elements, and roles, CDD enables a wider range of downstream tasks beyond simple judgment prediction, addressing a critical gap in available resources for this field.

What Problem Does This Paper Address?

The field of Legal Intelligence (LI), which aims to leverage AI for judicial services, has seen significant progress, particularly in areas like legal judgment prediction. However, the development of LI is severely hampered by a lack of diverse and large-scale datasets. Existing datasets often focus on specific tasks like judgment prediction and lack the richness needed for more complex analyses of judicial proceedings. This limits research into crucial areas like understanding the nuances of court debates, extracting detailed case information, and developing AI assistants that can truly support judges in their administrative and analytical tasks. The authors aim to bridge this gap by creating a comprehensive dataset that captures the dynamic nature of court trials.

Methods

The core contribution of this paper is the creation and annotation of the Court Debate Dataset (CDD). The data was collected from actual court trial records of private lending cases. To ensure the dataset's utility for various LI tasks, experienced judges were involved in defining key evidential features and factual elements crucial for determining case judgments. Law school students then annotated the dataset based on these definitions. The paper also defines five downstream tasks that can be addressed using CDD:

  • Fact Finding (FF): A text summarization task where the goal is to generate a summary of case facts from the dialogue, mirroring a judge's post-trial summarization.
  • Dialogue Generation (DG): A text generation task focused on generating a judge's utterance, potentially to assist judges in formulating questions or for simulating trial debates.
  • Feature Recognition (FR): A multi-label classification task to identify 12 predefined qualitative evidential features of a case from the dialogue.
  • Elements Identification (EI): A multi-label classification task to identify 14 predefined factual elements of a case from the dialogue.
  • Role Recognition (RR): A multi-class classification task to identify the speaker's role (judge, plaintiff, or defendant) for each utterance.

The authors also propose baseline models for these tasks, including traditional NLP models and large language models like LLaMA and LLaMA+SFT, to establish benchmark performance.

Datasets

The primary dataset introduced in this paper is the Court Debate Dataset (CDD).

  • CDD: Contains 30,481 court dialogue cases, totaling 1,144,425 utterances and 18,590,439 words. It is annotated with 12 qualitative features and 14 factual elements, and each utterance is tagged with the speaker's role. Sensitive information has been anonymized. The dataset is split into training, development, and test sets.
  • Comparison to Existing Datasets: The paper contrasts CDD with existing judicial datasets like CAIL, ECHR, and ILDC, highlighting that CDD's strength lies in its focus on multi-turn dialogues and its support for a broader range of tasks beyond just judgment prediction.

Key Insights and Ideas

  • Dialogue as a Rich Source: Court dialogues are a valuable and underutilized resource for Legal Intelligence, offering a dynamic and nuanced view of legal proceedings that goes beyond static case documents.
  • Expert-Driven Annotation: Involving experienced judges in defining features and elements is crucial for creating a dataset that is truly relevant and useful for judicial analysis.
  • Multi-Task Utility: CDD is designed to support a diverse set of NLP tasks, from summarization and generation to classification, making it a versatile benchmark for advancing LI.
  • LLMs Show Promise: Large Language Models, especially when fine-tuned (LLaMA+SFT), demonstrate superior performance across multiple tasks, indicating their potential for complex legal reasoning and analysis.

Key Results

The paper reports experimental results for the five defined tasks using various baseline models.

  • Fact Finding (Text Summarization): LLaMA+SFT achieved the best results, with ROUGE-L scores reaching 57.61, significantly outperforming traditional models.
  • Dialogue Generation: LLaMA+SFT also led in dialogue generation, achieving an ROUGE-L score of 54.84, demonstrating its capability to generate relevant judge utterances.
  • Text Classification (FR, EI, RR): LLaMA+SFT consistently outperformed other models, including BiLSTM and BERT, with macro-average F1 scores for Role Recognition reaching 77.20. This highlights the effectiveness of pre-trained LLMs for legal text classification tasks.

Limitations and Future Work

  • Limitations:
    • The dataset is primarily focused on private lending cases, which might limit its direct applicability to other legal domains without further adaptation.
  • While sensitive information is removed, the potential for residual bias in the data, inherited from real-world cases, is acknowledged.
  • Future Work:
    • The authors plan to continue developing new models to improve performance across the diverse sub-tasks supported by CDD.
  • They aim to expand the dataset to cover more legal domains and explore more complex LI applications.
  • Further research into addressing potential biases in the dataset and ensuring fairness in AI-driven judicial applications is implied.

Summary updated

BibTeX
@inproceedings{ji2023cdd,
  author = {Changzhen Ji and Yating Zhang and Adam Jatowt and Haipang Wu},
  title = {{CDD: A Large Scale Dataset for Legal Intelligence Research}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure