jurBERT: A Romanian BERT Model for Legal Judgement Prediction

Mihai Masala, Radu Cristian Alexandru Iacob, Ana Sabina Uban, Marina Cidota, Horia Velicu, Traian Rebedea, Marius PopescuNLLP @ EMNLP 2021 · OpenReview

Abstract

Researchers have introduced jurBERT, the first BERT-based model specifically pre-trained for the Romanian legal domain. By training on a massive 160GB corpus of court rulings, the model achieves state-of-the-art results in predicting legal outcomes, significantly outperforming general-purpose Romanian language models and classical machine learning baselines.

Introduction

While Transformer models like BERT have revolutionized Natural Language Processing (NLP), they often struggle in "low-resource" scenarios. This isn't just about rare languages; it also applies to highly specialized domains like law. Legal text is dense, uses archaic terminology, and follows rigid structural logic that general-purpose models (trained on Wikipedia or news) fail to grasp.

For the Romanian language, existing models like RoBERT were too generic for legal tasks. Furthermore, legal documents are often much longer than BERT’s 512-token limit, and real-world data is frequently "noisy"—containing OCR errors or irrelevant boilerplate text. The authors sought to bridge this gap by creating a domain-specific model and a methodology for applying it to messy, real-world legal cases.

Methods

The authors developed two variants: jurBERT-base and jurBERT-large. Unlike many projects that simply fine-tune an existing model, the team pre-trained jurBERT from scratch using Whole Word Masking (WWM).

To make the model useful for judgment prediction, they explored several specialized fine-tuning strategies:

  • Long Text Handling: Since legal cases exceed BERT's input limit, they tested different trimming methods. The winner was concatenating the first 512 tokens with the last 512 tokens of a document.
  • Optimization: They experimented with "slanted triangular learning rates" and "gradual layer unfreezing" to prevent the model from forgetting its legal knowledge during task-specific training.
  • Feature Engineering: For smaller, noisier datasets, they combined BERT’s deep embeddings with simple handcrafted features (like the year of the case or the county) to boost performance.

Datasets

The study utilized three distinct datasets, ranging from massive pre-training corpora to small, noisy real-world samples.

Summary of the datasets used for pre-training and downstream evaluation.
  1. RoJur (Pre-training): A massive collection of 11 million final rulings from Romanian civil courts (2010–2018), totaling 160GB of raw text.
  2. RoBanking (Downstream): A subset of 108,000 banking-related cases used to evaluate the model's ability to predict a verdict based on the plaintiff's and defendant's arguments.
  3. BRDCases (Real-world): A small (149 entries), highly specialized, and noisy dataset provided by a Romanian bank. This represented the "final boss" for the model, featuring OCR artifacts and very long lawsuit documents.

Key Results and Insights

jurBERT consistently outperformed RoBERT (the general Romanian BERT). This proves that for specialized fields, the "pre-train from scratch" approach is worth the massive computational cost if you have enough data.

Comparison of jurBERT and RoBERT performance on the RoBanking dataset across different strategies.
  • RoBanking Performance: jurBERT-base achieved an AUC of 86.73% when using pleas from both parties, setting a new benchmark for the task.
  • BRDCases Performance: On this difficult, small dataset, jurBERT-base (fine-tuned on RoBanking first) achieved an AUC of 59.65%. When the researchers added handcrafted features, the performance jumped to 61.46%, outperforming the SVM baseline (57.72%).
  • The "First + Last" Strategy: In legal documents, the most critical information is usually at the beginning (the claim) and the end (the conclusion). Simply taking the first 512 tokens is less effective than a "sandwich" approach that looks at both ends of the document.
  • Base vs. Large Paradox: Interestingly, while jurBERT-large performed better during pre-training, jurBERT-base often performed better on actual judgment prediction tasks. This suggests that larger models might be more prone to overfitting when the downstream dataset is relatively small.

Conclusion

The primary limitation remains the input length. Even with the "first + last" trimming strategy, the model still ignores the middle sections of very long documents, which might contain crucial evidence. Future work should explore "Long-sequence" Transformers like Longformer or Big Bird. Additionally, the authors aim to integrate more refined handcrafted features, such as the specific identity of the judge or the type of court, to further refine prediction accuracy in real-world scenarios.

Summary updated

BibTeX
@inproceedings{masala2021jurbert,
  author = {Mihai Masala and Radu Cristian Alexandru Iacob and Ana Sabina Uban and Marina Cidota and Horia Velicu and Traian Rebedea and Marius Popescu},
  title = {{jurBERT: A Romanian BERT Model for Legal Judgement Prediction}},
  booktitle = {NLLP @ EMNLP},
  year = {2021},
  url = {https://openreview.net/forum?id=hixkB428-rT},
}
Figure