Automated Extraction of Sentencing Decisions from Court Cases in the Hebrew Language

Mohr Wenger, Tom Kalir, Noga Berger, Carmit Klar Chalamish, Renana Keydar, Gabriel Stanovsky

Introduction

Presented at the 2021 Natural Legal Language Processing Workshop, this research introduces the first systematic approach to Automated Punishment Extraction (APE) from Hebrew criminal court cases. By focusing on sexual assault sentencing in Israel, the study combines legal domain expertise with Natural Language Processing (NLP) to transform unstructured judicial text into structured empirical data.

Legal systems generate vast quantities of unstructured text. However, extracting specific metrics—such as the exact duration of a prison sentence—remains a significant technical hurdle. This difficulty is amplified in Hebrew, a "low-resource" language compared to English, characterized by complex morphology and idiomatic judicial phrasing.

In the Israeli court system, sentencing decisions are written in natural language. A primary linguistic obstacle is the ambiguity of the Hebrew word for imprisonment (Ma’asar), which is used for both "actual" jail time and "conditional" (probationary) sentences. Accurately distinguishing between these is essential for empirical analysis of sentencing patterns and investigating judicial bias in high-stakes areas like sexual assault.

Methodology

The researchers developed a two-step pipeline to address the APE task:

  1. Imprisonment Sentence Detection: The system first identifies the specific sentence within a multi-page document that contains the final punishment. The study compared a rule-based approach—utilizing keyword scoring for "strong positive" verbs (e.g., sentencing) and "negative" indicators (e.g., docket numbers from previous cases)—against supervised machine learning models like Support Vector Machines (SVM) and Random Forests (RF).
  2. Duration Extraction: Once the relevant sentence is isolated, the system extracts the numerical duration. This involves regex-based identification tailored for Hebrew morphology and specific heuristics for "Combined Statements" (e.g., calculating actual time from a total that includes probation).
The two-step pipeline for identifying and extracting imprisonment durations from court decisions.

Dataset and Annotation

The study utilized a corpus of 1,043 sexual assault sentencing decisions from Israeli Magistrate and District Courts spanning 1990 to 2021, totaling approximately 3 million words. To establish a "Gold Standard" for evaluation, a subset of 100 cases (comprising 13,000 sentences) was manually annotated by legal scholars and practitioners.

Key Results and Insights

The evaluation revealed that in this linguistically complex and low-resource domain, traditional rule-based systems can outperform modern supervised models.

Comparison of model performance across sentence detection (Sent F1), automated punishment extraction (APE F1), and average error in months.

As shown in the results, the Rule-based model was the clear winner, achieving an APE F1 score of 0.65 and an average error of only 5 months. In contrast, the Random Forest and SVM models exhibited significantly higher average errors (11.6 and 10 months, respectively).

The superiority of the rule-based approach stems from its ability to incorporate "negative constraints" provided by legal experts. For instance, the model was programmed to ignore sentences containing brackets or backslashes (indicative of citations) or verbs in the past tense (referring to prior crimes). While supervised models were proficient at finding candidate sentences (high recall), they often struggled with "noise," failing to select the single correct sentence required for accurate extraction.

Conclusion and Future Directions

The study highlights the "Ma’asar" ambiguity as a primary failure point for general NLP models, where probationary periods are often mistaken for actual imprisonment. Using the optimized model on the full dataset, the researchers identified a median punishment of 36 months for sexual assault cases, though a significant portion (over one-third) fell below 15 months.

Future work aims to expand the annotated dataset and explore transformer-based architectures like AlephBERT to better capture the nuanced context of Hebrew legal language. Additionally, refining the system to better distinguish between "procedural" notes and actual sentencing pronouncements remains a priority for improving extraction accuracy.

Summary updated

BibTeX
@inproceedings{wenger2021automated,
  author = {Mohr Wenger and Tom Kalir and Noga Berger and Carmit Klar Chalamish and Renana Keydar and Gabriel Stanovsky},
  title = {{Automated Extraction of Sentencing Decisions from Court Cases in the Hebrew Language}},
  booktitle = {NLLP @ EMNLP},
  year = {2021},
}
Figure