JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences

David Beauchemin, Michelle Albert-Rochette, Richard Khoury, Pierre-Luc DézielEMNLP 2025

TL;DR

This paper introduces JUDGEBERT, a novel metric for evaluating how well simplified legal texts preserve the original meaning, a crucial but challenging task. They also present FrJUDGE, a new dataset of French insurance legal texts annotated for meaning preservation, and demonstrate that JUDGEBERT significantly outperforms existing metrics and passes critical sanity checks.

What Problem Does This Paper Address?

Automatic Text Simplification (ATS) aims to make complex texts easier to read while retaining their original meaning. However, evaluating this "meaning preservation" is tricky, especially in specialized domains like law. In legal contexts, even minor inaccuracies in simplified text can lead to significant misunderstandings, legal issues, and create a "legal gap" between the original and simplified versions. Existing ATS evaluation metrics, like BLEU and SARI, often focus on only one aspect (fluency or simplicity) and don't specifically address the nuances of legal meaning. Furthermore, current Transformer-based metrics, while better, haven't been benchmarked against human judgment for legal meaning preservation, nor have they been specifically designed for French legal texts. This paper tackles this gap by proposing a new metric and dataset tailored for French legal text simplification.

Methods

The core contribution is the JUDGEBERT metric, a fine-tuned BERT-based model designed to assess legal meaning preservation. It's built upon the CamemBERT-baseV2 model and uses a regression head to predict a score. The authors also define "legal meaning" as how well the output text conveys legal details and exceptions without misrepresenting the law. To train and evaluate JUDGEBERT, they introduce FrJUDGE, a new dataset of 297 human-annotated French insurance legal sentences. The annotation process involved native French-speaking law students who assessed simplicity, characterization, and legal meaning preservation using a Likert scale. They also conducted two automated sanity checks: ensuring identical sentences score 100% and unrelated sentences score 0%, a crucial step where many existing metrics fail.

Datasets

  • FrJUDGE: A newly introduced dataset consisting of 297 human-annotated French insurance legal sentences. Each instance includes the original legal sentence, its simplified version, and annotations for simplicity, characterization, and legal meaning. This is the first legal meaning judgment dataset in any language.
  • Existing French Legal Text Simplification Corpora: The paper briefly mentions RISCBAC and EUR-Lex-Sum as existing French legal corpora, but notes they lack simplifications or human annotations suitable for their task.

Key Insights and Ideas

  • "Legal Meaning" is Distinct: The paper argues that "legal meaning" is not the same as general "meaning preservation." Synonyms that are interchangeable in everyday language might carry different legal weight (e.g., "automobile" vs. "vehicle").
  • Existing Metrics Fall Short: Many standard ATS metrics (BLEU, SARI) are insufficient for legal text. Even advanced Transformer-based metrics show weak correlation with human judgment for legal meaning preservation and fail basic sanity checks.
  • Sanity Checks are Crucial: The proposed sanity checks (identical sentences score 100%, unrelated score 0%) are vital for establishing a metric's credibility, especially in a high-stakes domain like law. JUDGEBERT, particularly when trained with data augmentation, passes these checks, unlike many others.
  • Human Judgment is Key, but Costly: While human evaluation is the gold standard, it's expensive and time-consuming. The goal is to develop automatic metrics that can reliably approximate human judgment.
  • Data Augmentation Improves Performance: Training JUDGEBERT with augmented data (combining FrJUDGE with sanity-check data) significantly improved its performance on the sanity checks, suggesting the benefit of exposing the model to more diverse examples.

Key Results

  • JUDGEBERT demonstrated a superior correlation with human judgment compared to existing Transformer-based metrics, achieving a near-perfect Pearson correlation (0.97) when trained with data augmentation (JUDGEBERT-DA).
  • JUDGEBERT-DA passed both sanity checks (100% for identical sentences, 0% for unrelated sentences), a feat not achieved by most other metrics.
  • Other metrics showed low to moderate correlations with human judgment and often failed the sanity checks, indicating their inadequacy for legal meaning preservation. For instance, many metrics returned non-zero scores for unrelated sentences, highlighting their inability to grasp legal nuances without specific domain knowledge.

Limitations and Future Work

  • Limited Domain Scope: JUDGEBERT was trained on French insurance texts. Its generalization to other legal domains or languages is an open question.
  • Dataset Size: The FrJUDGE dataset, while novel, is relatively small for training large language models.
  • Potential Overfitting: The authors acknowledge the risk of overfitting due to the small training dataset.
  • Contextual Limitations: The FrJUDGE corpus consists of isolated sentences, which differs from how legal documents are typically analyzed in their full context.
  • Future Work: The authors plan to explore JUDGEBERT's generalization to other languages and tasks, expand FrJUDGE with more insurance products and jurisdiction-agnostic clauses, and investigate approaches that consider the full document context for simplification. They also hope their work will spur further development of robust metrics for legal NLP.

Summary updated

BibTeX
@inproceedings{beauchemin2025judgebert,
  author = {David Beauchemin and Michelle Albert-Rochette and Richard Khoury and Pierre-Luc Déziel},
  title = {{JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure