A Survey of Classification Tasks and Approaches for Legal Contracts

Amrita Singh, Aditya Joshi, Jiaojiao Jiang, Hye-young Paik

Abstract

Given the large size and volumes of contracts and their underlying inherent complexity, manual reviews become inefficient and prone to errors, creating a clear need for automation. Automatic Legal Contract Classification (LCC) revolutionizes the way legal contracts are analyzed, offering substantial improvements in speed, accuracy, and accessibility. This survey delves into the challenges of automatic LCC and a detailed examination of key tasks, datasets, and methodologies. We identify seven classification tasks within LCC, and review fourteen datasets related to English-language contracts, including public, proprietary, and non-public sources. We also introduce a methodology taxonomy for LCC, categorized into Traditional Machine Learning, Deep Learning, and Transformer-based approaches. Additionally, the survey discusses evaluation techniques and highlights the best-performing results from the reviewed studies. By providing a thorough overview of current methods and their limitations, this survey suggests future research directions to improve the efficiency, accuracy, and scalability of LCC. As the first comprehensive survey on LCC, it aims to support legal NLP researchers and practitioners in improving legal processes, making legal information more accessible, and promoting a more informed and equitable society.

Legal contracts are the backbone of commerce, but they are notoriously difficult to process. They are written in "Legalese"—a formal, complex language characterized by nested clauses, archaic terminology, and intricate cross-references. Manual review is slow, expensive, and prone to human error, creating a massive bottleneck for businesses and a barrier to "access to justice" for individuals.

An example of a software license agreement highlighting the complexities of 'Legalese' such as nested clauses and cross-references.

While the broader field of Legal NLP has grown, the authors argue that Contractual NLP has been underserved. Previous surveys often lumped contracts in with court rulings or legislation, ignoring the unique structural and semantic challenges of agreements. This paper fills that gap by providing a roadmap for researchers and engineers specifically interested in automating the scrutiny of legal agreements.

Methodology Taxonomy

The authors categorize LCC methodologies into three historical and technological waves:

  1. Classical Machine Learning: Early approaches relied on heavy feature engineering. Techniques like Bag-of-Words (BoW), TF-IDF, and N-grams were paired with classifiers like Support Vector Machines (SVM) and Random Forests. These are now primarily used as robust baselines.
  2. Classical Deep Learning: The introduction of Recurrent Neural Networks (RNNs), specifically BiLSTMs with attention mechanisms, allowed models to capture long-term dependencies in text. Convolutional Neural Networks (CNNs) were also adapted to identify specific patterns within clauses.
  3. Transformer-based Methods: This represents the current state-of-the-art. The survey breaks these down into pre-training and fine-tuning (e.g., Legal-BERT), prompting with LLMs (GPT-4, LLaMA), model compression, and data augmentation for rare clause types.

Positioning the Survey

This survey distinguishes itself by focusing specifically on Legal Contract Classification (LCC). As shown in the comparison below, while previous surveys covered various legal NLP tasks, they often lacked in-depth coverage of contractual nuances or the latest transformer-based methodologies applied specifically to LCC.

Comparison of this survey with previous legal NLP literature, highlighting its unique focus on Contractual NLP and Legal Contract Classification (LCC).

Datasets and Key Insights

The survey reviews 14 datasets, categorized by the tasks they support. Key highlights include LEDGAR, a massive multi-label corpus of provisions from SEC filings; CUAD, an expert-annotated dataset of commercial contracts; and UNFAIR-ToS, focused on detecting unfair clauses in online Terms of Service.

Several key insights emerged from the analysis:

  • Domain Adaptation is King: Models pre-trained on legal text (Legal-BERT, CaseLaw-BERT) consistently outperform their "vanilla" counterparts.
  • The Prompting Paradox: While LLMs show impressive zero-shot reasoning, smaller, fine-tuned encoder models (like DeBERTa) often provide more reliable results for specialized tasks.
  • The "Legalese" Bottleneck: Long-range dependencies (where a definition on page 1 affects a clause on page 50) remain a significant technical hurdle for standard NLP architectures.
  • Class Imbalance: Finding "red flag" clauses requires specialized loss functions or data augmentation due to the rarity of such clauses in standard contracts.

Key Results and Future Directions

Transformer-based models on the LEDGAR dataset achieve Micro-F1 scores ranging from 67% to 88.3%, significantly outperforming SVM baselines. On the UNFAIR-ToS dataset, the best models reach Micro-F1 scores between 64.7% and 96.3%. Furthermore, research into model compression showed that models could achieve up to a 9.4x speedup in inference with minimal loss in accuracy.

The authors identify several critical "white spaces" for future research, including addressing jurisdictional bias (expanding beyond U.S. and EU law), improving explainability (XAI) through neuro-symbolic AI, tackling small data challenges with few-shot learning, and developing multi-task learning models that can simultaneously classify topics and detect risks.

Summary updated

Abstract

Given the large size and volumes of contracts and their underlying inherent complexity, manual reviews become inefficient and prone to errors, creating a clear need for automation. Automatic Legal Contract Classification (LCC) revolutionizes the way legal contracts are analyzed, offering substantial improvements in speed, accuracy, and accessibility. This survey delves into the challenges of automatic LCC and a detailed examination of key tasks, datasets, and methodologies. We identify seven classification tasks within LCC, and review fourteen datasets related to English-language contracts, including public, proprietary, and non-public sources. We also introduce a methodology taxonomy for LCC, categorized into Traditional Machine Learning, Deep Learning, and Transformer-based approaches. Additionally, the survey discusses evaluation techniques and highlights the best-performing results from the reviewed studies. By providing a thorough overview of current methods and their limitations, this survey suggests future research directions to improve the efficiency, accuracy, and scalability of LCC. As the first comprehensive survey on LCC, it aims to support legal NLP researchers and practitioners in improving legal processes, making legal information more accessible, and promoting a more informed and equitable society.
BibTeX
@inproceedings{singh2025survey,
  author = {Amrita Singh and Aditya Joshi and Jiaojiao Jiang and Hye-young Paik},
  title = {{A Survey of Classification Tasks and Approaches for Legal Contracts}},
  booktitle = {Artif. Intell. Rev},
  year = {2025},
  url = {https://openreview.net/forum?id=IXyvJ1EfqN},
}
Figure