Detecting Relevant Differences Between Similar Legal Texts

Xiang Li, Jiaxun Gao, Diana Inkpen, Wolfgang AlschnerNLLP @ EMNLP 2022 · DOI

TL;DR

This paper tackles the challenge of automatically identifying semantically relevant differences between highly similar legal texts, a crucial task for legal professionals. They introduce a novel deep learning approach that leverages contextual information and a carefully constructed dataset derived from international investment treaties to distinguish truly important distinctions from mere stylistic variations.

What Problem Does This Paper Address?

Legal documents, especially in areas like international investment treaties, often share a common structure and boilerplate language. While this standardization is useful, lawyers and researchers are primarily interested in the subtle yet significant textual differences that signal deviations from norms, industry standards, or model laws. Traditional text similarity measures, like cosine similarity on TF-IDF vectors, struggle to capture these nuanced semantic differences, often being misled by stylistic variations or failing to detect small but important changes. Existing methods for detecting differences are often too simplistic and don't account for the legal relevance of these distinctions. This paper aims to bridge this gap by developing a model that can pinpoint differences that matter legally, not just those that are syntactically or lexically distinct.

Methods

The core of this work lies in developing a deep learning model for detecting relevant differences. The authors propose a two-stage approach:

  1. Structure Category Prediction: As a preliminary step, they address the challenge of predicting article structure categories (e.g., Preamble, Definitions) for articles that lack manual metadata. This is achieved using a BERT-based model that leverages contextual information by considering both the article in question and its succeeding article within the treaty. This helps improve classification accuracy, especially for articles without explicit titles.
  2. Relevant Difference Detection: For the main task, they employ a BERT-based model. The key innovation here is the data preprocessing and model architecture.
  3. Data Preprocessing: They introduce a novel method for automatically labeling sentence pairs. This involves using existing metadata from the Electronic Database of Investment Treaties (EDIT) to identify highly similar sentences within the same structure category. They then use content categories (derived from keywords) to label pairs as "stylistic" (sharing content categories) or "relevant" (differing content categories). They also include "irrelevant" differences for robustness. A crucial preprocessing step involves removing a significant portion (85%) of categorical keywords, which they found improves model generalization and prevents overfitting to specific terms.
  4. Model Architecture: While specific details of the difference detection model architecture are not fully elaborated in the provided text, it's implied to be a BERT-based approach that takes pairs of sentences as input. The paper mentions fine-tuning BERT models and experimenting with different input constructions.

Datasets

The primary dataset used in this research is the Electronic Database of Investment Treaties (EDIT). This is a comprehensive, manually annotated resource of international investment agreements. The authors leverage its existing metadata (structure categories, content categories derived from keywords) to automatically construct their training and testing datasets for the relevant difference detection task. They created a dataset of 12,968 sentence pairs, categorized into "stylistic," "relevant," and "irrelevant" differences.

Key Insights and Ideas

  • Semantic Relevance is Key: The paper strongly emphasizes that for legal professionals, the semantic relevance of a difference is paramount, not just its mere existence. A change in wording that alters the legal meaning or implication is what matters.
  • Context Matters for Structure Prediction: Using the succeeding article as context significantly improves the accuracy of predicting article structure categories, highlighting the interconnectedness of legal text.
  • Keyword Removal for Generalization: Removing a substantial portion of categorical keywords from the input significantly improves the model's ability to generalize and avoid overfitting, a counter-intuitive but effective strategy.
  • Automatic Labeling is Feasible: The research demonstrates that it's possible to automatically generate a labeled dataset for this complex task by cleverly utilizing existing legal metadata, reducing the need for extensive manual annotation.

Key Results

  • Structure Category Prediction: The BERT-base-A model (using auxiliary succeeding article) achieved the highest accuracy (0.963) for structure category prediction, outperforming baseline models and a BERT-base-S model.
  • Relevant Difference Detection: The ROBERTa model achieved the best performance for relevant difference detection, with an F1 score of 0.939 and an accuracy of 0.960. This suggests that advanced transformer models are highly effective for this task.
  • LegalBERT Performance: Interestingly, the domain-specific LegalBERT model lagged behind the general-purpose RoBERTa model, indicating that while domain-specific pre-training is beneficial, general-purpose models fine-tuned on the specific task can also be very competitive.
  • Keyword Removal Effectiveness: Experiments showed that removing 85% of categorical keywords led to better results (e.g., 0.85 F1 for CNN FastText) compared to keeping all keywords (0.78 F1) or removing all keywords (0.71 F1), validating the preprocessing strategy.

Limitations and Future Work

  • Limitations:
    • The research is primarily focused on articles from legal treaties. While the techniques are potentially applicable to other legal texts or general language, the lack of annotated data for other domains is a significant barrier.
  • The dataset for structure category prediction is imbalanced, and existing re-sampling methods are not ideal for highly standardized legal articles.
  • Future Work:
    • Applying text entailment methods to articles identified as having relevant differences to explore historical evolution of treaty writing.
  • Investigating domain-specific re-sampling methods for imbalanced datasets.
  • Extending the techniques to other types of legal texts and general language domains, contingent on the availability of annotated data.

Summary updated

BibTeX
@inproceedings{li2022detecting,
  author = {Xiang Li and Jiaxun Gao and Diana Inkpen and Wolfgang Alschner},
  title = {{Detecting Relevant Differences Between Similar Legal Texts}},
  booktitle = {NLLP @ EMNLP},
  year = {2022},
  doi = {10.18653/v1/2022.nllp-1.24},
}
Figure