Introduction
Presented at the 2021 Natural Legal Language Processing Workshop, this research introduces the first systematic approach to Automated Punishment Extraction (APE) from Hebrew criminal court cases. By focusing on sexual assault sentencing in Israel, the study combines legal domain expertise with Natural Language Processing (NLP) to transform unstructured judicial text into structured empirical data.
The Challenge of Hebrew Legal NLP
Legal systems generate vast quantities of unstructured text. However, extracting specific metrics—such as the exact duration of a prison sentence—remains a significant technical hurdle. This difficulty is amplified in Hebrew, a "low-resource" language compared to English, characterized by complex morphology and idiomatic judicial phrasing.
In the Israeli court system, sentencing decisions are written in natural language. A primary linguistic obstacle is the ambiguity of the Hebrew word for imprisonment (Ma’asar), which is used for both "actual" jail time and "conditional" (probationary) sentences. Accurately distinguishing between these is essential for empirical analysis of sentencing patterns and investigating judicial bias in high-stakes areas like sexual assault.
Methodology
The researchers developed a two-step pipeline to address the APE task:
- Imprisonment Sentence Detection: The system first identifies the specific sentence within a multi-page document that contains the final punishment. The study compared a rule-based approach—utilizing keyword scoring for "strong positive" verbs (e.g., sentencing) and "negative" indicators (e.g., docket numbers from previous cases)—against supervised machine learning models like Support Vector Machines (SVM) and Random Forests (RF).
- Duration Extraction: Once the relevant sentence is isolated, the system extracts the numerical duration. This involves regex-based identification tailored for Hebrew morphology and specific heuristics for "Combined Statements" (e.g., calculating actual time from a total that includes probation).
Dataset and Annotation
The study utilized a corpus of 1,043 sexual assault sentencing decisions from Israeli Magistrate and District Courts spanning 1990 to 2021, totaling approximately 3 million words. To establish a "Gold Standard" for evaluation, a subset of 100 cases (comprising 13,000 sentences) was manually annotated by legal scholars and practitioners.
Key Results and Insights
The evaluation revealed that in this linguistically complex and low-resource domain, traditional rule-based systems can outperform modern supervised models.
As shown in the results, the Rule-based model was the clear winner, achieving an APE F1 score of 0.65 and an average error of only 5 months. In contrast, the Random Forest and SVM models exhibited significantly higher average errors (11.6 and 10 months, respectively).
The superiority of the rule-based approach stems from its ability to incorporate "negative constraints" provided by legal experts. For instance, the model was programmed to ignore sentences containing brackets or backslashes (indicative of citations) or verbs in the past tense (referring to prior crimes). While supervised models were proficient at finding candidate sentences (high recall), they often struggled with "noise," failing to select the single correct sentence required for accurate extraction.
Conclusion and Future Directions
The study highlights the "Ma’asar" ambiguity as a primary failure point for general NLP models, where probationary periods are often mistaken for actual imprisonment. Using the optimized model on the full dataset, the researchers identified a median punishment of 36 months for sexual assault cases, though a significant portion (over one-third) fell below 15 months.
Future work aims to expand the annotated dataset and explore transformer-based architectures like AlephBERT to better capture the nuanced context of Hebrew legal language. Additionally, refining the system to better distinguish between "procedural" notes and actual sentencing pronouncements remains a priority for improving extraction accuracy.
Summary updated