Validity Assessment of Legal Will Statements as Natural Language Inference

Alice Saebom Kwak, Jacob O. Israelsen, Clayton T. Morrison, Derek E. Bambauer, Mihai Surdeanu

Abstract

The probate process—the legal validation and execution of a will—is a complex, resource-intensive procedure prone to human error. As electronic wills become more prevalent, the need for automated verification tools grows. This paper explores the application of Natural Language Inference (NLI) to assess the validity of legal will statements. We introduce a novel dataset based on Tennessee law and propose a "three-tuple" input framework—comprising the will statement, the relevant law, and the factual conditions at the time of death. Our results demonstrate that while modern transformer models achieve high standard accuracy, they struggle with consistent logical reasoning when evaluated using stricter metrics like Group Accuracy (GA).

Introduction

Natural Language Inference (NLI) is traditionally framed as a binary relationship where a model determines if a "premise" entails or contradicts a "hypothesis." However, legal validity is rarely a simple relationship between two sentences. In the context of a last will and testament, a clause's validity is contingent upon both the governing legal code and the specific factual circumstances existing at the time of the testator's death. For example, a clause appointing an executor is only valid if that individual meets statutory requirements (e.g., lack of a felony conviction) at the time of probate.

This study addresses the gap between standard NLI and the multi-factor reasoning required for legal analysis. By framing legal validity as a three-body problem involving the will statement, the law, and the condition, we provide a more robust framework for automated legal reasoning.

Methodology

To accommodate the complexity of legal reasoning, we adapted standard neural architectures to process a three-tuple input. Each input instance consists of:

  1. Statement: The specific clause extracted from the will.
  2. Law: The relevant section of the Tennessee Code.
  3. Condition: Hypothetical facts or circumstances at the time of probate.

To facilitate model processing, these segments were concatenated and prefixed with special tokens: [STATE], [LAW], and [COND]. We evaluated eight neural models, including standard Transformers (BERT, RoBERTa), the Longformer (designed for long-sequence processing), and Sentence-Transformers utilizing the SetFit (Sentence Transformer Fine-Tuning) framework.

Evaluation Metrics

Beyond standard accuracy and F1-scores, we introduced Group Accuracy (GA) to measure the consistency of model reasoning. GA groups all positive and negative variations derived from a single will statement. A model receives credit for a group only if it correctly predicts every variation within that group. This metric is designed to expose whether a model has truly captured the underlying legal logic or is merely relying on superficial pattern matching.

Dataset

We developed a new, open-access dataset derived from 23 publicly available Tennessee wills dated post-1970. The dataset contains 1,014 annotated data points. Unlike standard NLI datasets like SNLI, which feature short sentences (8–14 tokens), our dataset features significantly higher linguistic complexity, with an average length of 269 tokens per instance. Labels were categorized as support, refute, or unrelated.

Results and Analysis

Our experiments yielded several key insights into the capabilities and limitations of current LLMs in the legal domain.

Performance Benchmarks

roberta-large-mnli emerged as the top-performing model, achieving a standard accuracy of 96.86% and an F1-score of 96.42. Interestingly, this model outperformed longformer-base-4096, despite the latter's ability to handle longer sequences without truncation. This suggests that the most critical information for determining legal validity is often concentrated in specific segments that standard models can effectively capture even when the full text exceeds their token limits.

The Reasoning Gap

While standard accuracy was high, Group Accuracy (GA) revealed a significant performance drop. The best model’s GA was 84.31%, representing a ~12% gap compared to its standard accuracy. This discrepancy indicates that models are often inconsistent; they may correctly identify validity under one set of conditions but fail when the factual conditions are slightly altered, despite the underlying legal logic remaining the same.

Ablation and Explainability

Ablation studies confirmed the necessity of the three-tuple approach. Removing either the "Law" or "Condition" segments resulted in a catastrophic drop in performance. Notably, models appeared to rely more heavily on the "Law" segment than the "Condition" segment. Furthermore, using LIME (Local Interpretable Model-agnostic Explanations), we observed that models occasionally based their predictions on non-substantive "garbage" tokens (e.g., "and", "the") rather than legal keywords, suggesting a risk of overfitting to the stylistic nuances of legal prose.

Discussion and Limitations

The primary limitation of this research is its geographic and sample scope. The dataset is currently limited to Tennessee law; because legal statutes vary significantly across jurisdictions, the models' generalizability to other states remains unproven. Additionally, the current study lacks human-in-the-loop validation from legal experts to verify the "reasoning" paths identified by explainability tools.

Conclusion

This work demonstrates that while NLI is a promising path for automating legal validity assessments, standard metrics may overestimate a model's true understanding. The three-tuple input framework provides a necessary foundation for legal reasoning, but the gap in Group Accuracy highlights the need for more sophisticated architectures that can move beyond pattern matching toward deep, consistent logical inference. Future work will focus on expanding the dataset to multiple jurisdictions and involving legal professionals in the model training and validation process.

Summary updated

BibTeX
@inproceedings{kwak2022validity,
  author = {Alice Saebom Kwak and Jacob O. Israelsen and Clayton T. Morrison and Derek E. Bambauer and Mihai Surdeanu},
  title = {{Validity Assessment of Legal Will Statements as Natural Language Inference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.447},
}
Figure