Annotating Topical Legal Insights from Case Proceedings

Subinay Adhikary, Dwaipayan Roy, Debasis Ganguly, Shouvik Kumar Guha, Kripabandhu GhosharXiv 2026 · arXiv

TL;DR

The authors introduce LeDA (Legal Data Annotation), an open-source, web-based tool designed to capture complex, fine-grained thematic concepts in legal documents. Unlike traditional annotation software, LeDA supports dynamic tag creation, multi-labeling, and built-in inter-annotator agreement calculations, making it uniquely suited for building structured datasets from unstructured legal text.

What Problem Does This Paper Address?

Extracting meaningful information from legal case proceedings is notoriously difficult. Standard Natural Language Processing (NLP) tasks often focus on Named Entity Recognition (NER)—identifying atomic entities like names, dates, or locations. However, understanding a legal case requires capturing broader thematic "aboutness" (e.g., identifying that a sequence of events constitutes a "murder on parole" or a "property dispute").

Existing text annotation tools fall short in the legal domain for several reasons:

  1. Lack of Pre-defined Ontologies: Legal concepts are often discovered organically as an expert reads through documents, meaning a static, pre-defined set of labels is insufficient.
  2. Complex Spans: Legal concepts frequently cross sentence boundaries and require multiple overlapping tags.
  3. Adjudication Hurdles: Resolving disagreements between legal experts (meta-annotation) is challenging without specialized dual-view interfaces and custom Inter-Annotator Agreement (IAA) metrics.

Methods

To address these gaps, the authors developed LeDA, a web-based annotation system built with a Python Django backend and a standard HTML/CSS/JS frontend.

LeDA introduces several specialized features tailored for legal workflows:

  • Dynamic Tagging: Annotators can request and create new tags on the fly when encountering fine-grained information not covered by the initial schema.
  • Multi-Tagging & Highlighting: Users can highlight multi-sentence text spans and assign multiple overlapping labels to capture nuanced legal contexts.
  • Super-Annotator View & Adjudication: A dedicated interface allows a senior expert to view conflicting annotations side-by-side and reconcile differences.
  • Custom IAA Calculation: Because legal annotations involve overlapping spans and multiple tags, LeDA implements a novel IAA computation method that factors in the exact text span, the assigned tags, and the sentence numbers.

The authors compared LeDA against popular general-purpose annotation tools, highlighting its comprehensive feature set:

Table 2. Feature-wise comparison between different tools.

The complete user workflow—from uploading documents to resolving low IAA scores—is managed within a single unified interface:

LeDA system workflow interface.

Datasets

The system was evaluated by annotating 200 legal case documents from the Indian Supreme Court (sourced via Indian Kanoon). The annotation team consisted of three legal practitioners from the West Bengal National University of Juridical Sciences (two primary annotators and one super-annotator).

Key Insights and Ideas

  • Legal Concepts Transcend Sentences: The authors observed that legal concepts vary drastically in length. For instance, a Homicide_murder tag might span a single sentence, whereas fully capturing Expert_witness_testimony typically requires 4 to 5 sentences.
  • Ontologies Must Evolve: Forcing annotators to use a rigid, static schema limits discovery. Allowing legal experts to dynamically add tags (such as distinguishing an Investigation_agency like the CBI or NIA) significantly improves the richness of the final dataset.

The initial static schema and the dynamically added expert tags are detailed below:

Table 1. A set of tags and their descriptions used in LeDA.
  • Nuanced Overlaps: A single event often embodies multiple legal dimensions simultaneously. As shown in the paper's case study, a single headnote can be tagged as both Murder_on_parole and Second_murder to accurately preserve its thematic meaning for downstream ML tasks.

Key Results

  • Successful Deployment: Legal experts successfully utilized LeDA to annotate the 200-document corpus.
  • High User Satisfaction: Feedback from the legal practitioners indicated that the tool met all operational requirements, with no additional feature requests or modifications suggested.
  • Downstream Readiness: The resulting dataset represents documents as structured "bags of concepts" rather than flat text, paving the way for more advanced downstream applications like automated judgment prediction and prior case retrieval.

Limitations and Future Work

While LeDA proves highly effective, the current dataset and case studies are primarily restricted to murder-related case proceedings. The authors acknowledge the need to expand and validate the tool across other legal domains (e.g., civil, corporate, or tax law).

Additionally, manual annotation by legal experts remains expensive and time-consuming. The authors plan to use the meticulously annotated dataset generated by LeDA to train NLP models for automated concept extraction, ultimately reducing the reliance on manual labor.

Summary updated

Abstract

In this paper, we mainly concentrate on finding concepts or topics from the legal case proceedings, since adopting a structured representation for legal documents, as opposed to a mere bag-of-words flat text representation, can significantly enhance processing capabilities. To achieve this objective, we put forward a set of diverse concepts for legal case proceedings. With this motivation, we propose LeDA, a system for Legal Data Annotation. The system offers the generic functionality of annotating and adjudicating entities or concepts within documents via a web-based interface. A novel feature of our system is that it allows to dynamic create new tags for annotation, which is a particularly useful provision for situations where there exists no pre-defined ontology for the entities (concepts) that need to be annotated - these being rather discovered by annotators as they continue examining more documents. The system that we demonstrate is currently in use to annotate a set of concepts from legal documents to construct semantic representations of documents as bags of concepts that can then be used for several downstream tasks, such as prior case retrieval, judgment prediction, and so on. Along with the system features in general, we also describe how LeDA was used by 3 assessors to annotate and adjudicate legal concept names from Indian Supreme Court case proceedings.
BibTeX
@misc{adhikary2026annotating,
  author = {Subinay Adhikary and Dwaipayan Roy and Debasis Ganguly and Shouvik Kumar Guha and Kripabandhu Ghosh},
  title = {{Annotating Topical Legal Insights from Case Proceedings}},
  year = {2026},
  eprint = {2607.27792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.27792},
}
Figure