LLMs for Law: Evaluating Legal-Specific LLMs on Contract Understanding

Amrita Singh, H. Suhan Karaca, Aditya Joshi, Hye-young Paik, Jiaojiao Jiang

Abstract

Despite advances in legal NLP, no comprehensive evaluation covering multiple legal-specific LLMs currently exists for contract classification tasks in contract understanding. To address this gap, we present an evaluation of 10 legal-specific LLMs on three English language contract understanding tasks and compare them with 7 general-purpose LLMs. The results show that legal-specific LLMs consistently outperform general-purpose models, especially on tasks requiring nuanced legal understanding. Legal-BERT and Contracts-BERT establish new SOTAs on two of the three tasks, despite having 69% fewer parameters than the best-performing general-purpose LLM. We also identify CaseLaw-BERT and LexLM as strong additional baselines for contract understanding. Our results provide a holistic evaluation of legal-specific LLMs and will facilitate the development of more accurate contract understanding systems.

Introduction

The legal domain presents a unique challenge for natural language processing due to "legalese"—a specialized dialect characterized by complex syntax and precise semantic requirements. While numerous legal-specific Large Language Models (LLMs) like Legal-BERT and CaseLaw-BERT have been developed, they are frequently bypassed in favor of massive general-purpose models such as RoBERTa-large or GPT-3.5. This study addresses a critical gap in the literature: the lack of a systematic, head-to-head evaluation to determine if domain-specialized models provide a tangible performance advantage in contract understanding tasks compared to their larger, general-purpose counterparts.

Methodology

The evaluation employed Supervised Fine-Tuning (SFT) across 17 distinct models:

  • 10 Legal-Specific Models: Including Legal-BERT, Contracts-BERT, Legal-RoBERTa, and LexLM, pre-trained on diverse legal corpora (e.g., US contracts, Indian court cases).
  • 7 General-Purpose Models: Including BERT, RoBERTa (base and large), DeBERTa, and Longformer.
  • Prompting Baselines: Zero-shot and few-shot performance of GPT-3.5-Turbo was also assessed for comparison.

Performance was measured using micro-F1 and macro-F1 scores to account for the class imbalances inherent in legal datasets.

Datasets

The study utilized three publicly available English-language contract datasets, each targeting a different aspect of contract understanding:

  1. UNFAIR-ToS: Identification of unfair clauses in Terms of Service (e.g., unilateral termination rights).
  2. LEDGAR: Classification of contract provision topics across 100 categories (e.g., "Governing Laws" or "Warranties").
  3. LEXDEMOD: Detection of deontic modality—determining if a clause specifies an obligation, permission, or prohibition for a specific party.
Summary of the three contract understanding datasets used in the evaluation.

Results and Analysis

The experimental results reveal a "David vs. Goliath" scenario where smaller, specialized models consistently outperform larger general-purpose models in nuanced legal classification.

Performance comparison of general-purpose and legal-specific LLMs across three contract understanding tasks.

Domain Specificity vs. Scale

The most significant finding is that Legal-BERT and Contracts-BERT (both ~110M parameters) established new State-of-the-Art (SOTA) results on two of the three tasks. Notably, they outperformed RoBERTa-large (355M parameters) despite having 69% fewer parameters. This suggests that for legal tasks, domain-specific pre-training is more impactful than raw parameter count.

Task-Specific Performance

  • UNFAIR-ToS: Contracts-BERT achieved the highest macro-F1 (83.4), surpassing RoBERTa-large (81.6). This highlights the benefit of pre-training on contract-specific text.
  • LEXDEMOD: Legal-BERT set a new SOTA with a macro-F1 of 78.01, compared to the general-purpose baseline of 77.88.
  • LEDGAR: General-purpose models (RoBERTa-large) performed competitively here, likely because topic classification relies more on keyword recognition than deep semantic legal reasoning. However, Legal-BERT still delivered nearly equivalent performance.
  • GPT-3.5-Turbo: Prompted models lagged significantly behind fine-tuned smaller models, reinforcing that SFT on domain-specific encoders remains the superior architecture for specialized legal tasks.

Discussion and Limitations

The study demonstrates that specialized pre-training provides an "internal dictionary" for legal semantics that general models lack. However, newer models do not always guarantee better performance; for instance, PoL-BERT underperformed, likely due to a pre-training corpus that was less relevant to commercial contracts.

Limitations: * Language Scope: The evaluation was restricted to English-language contracts. * Architectural Focus: The study primarily focused on encoder-only models. Preliminary results for encoder-decoder models like LexT5 showed lower performance in these classification tasks. * Corpus Diversity: Many legal LLMs are trained on court cases, which may not perfectly translate to the structural nuances of commercial contracts.

Future research will aim to expand this benchmark to multilingual legal tasks and evaluate emerging decoder-based legal models as they become integrated into standard evaluation frameworks.

Summary updated

Abstract

Despite advances in legal NLP, no comprehensive evaluation covering multiple legal-specific LLMs currently exists for contract classification tasks in contract understanding. To address this gap, we present an evaluation of 10 legal-specific LLMs on three English language contract understanding tasks and compare them with 7 general-purpose LLMs. The results show that legal-specific LLMs consistently outperform general-purpose models, especially on tasks requiring nuanced legal understanding. Legal-BERT and Contracts-BERT establish new SOTAs on two of the three tasks, despite having 69% fewer parameters than the best-performing general-purpose LLM. We also identify CaseLaw-BERT and LexLM as strong additional baselines for contract understanding. Our results provide a holistic evaluation of legal-specific LLMs and will facilitate the development of more accurate contract understanding systems.
BibTeX
@misc{singh2025llms,
  author = {Amrita Singh and H. Suhan Karaca and Aditya Joshi and Hye-young Paik and Jiaojiao Jiang},
  title = {{LLMs for Law: Evaluating Legal-Specific LLMs on Contract Understanding}},
  year = {2025},
  eprint = {2508.07849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.07849},
}
Figure