Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Models

Stelios Maroudas, Sotiris Legkas, Prodromos Malakasiotis, Ilias ChalkidisarXiv 2022 · arXiv

This "open diary" from a legal-tech R&D team provides a practical roadmap for building domain-specific, multilingual language models that are small enough for production but powerful enough to handle complex legal reasoning. By combining scratch-built pre-training with a "gradual compression" pipeline, the authors demonstrate how to achieve a 66x inference speedup with minimal performance loss.

TL;DR

The researchers developed C-XLM, a suite of multilingual legal-oriented language models, and introduced a Gradual Compression (GC) pipeline that combines pruning, distillation, and quantization. Their results show that highly compressed models can outperform standard fine-tuned small models, offering a path to deploy state-of-the-art legal NLP on modest hardware without breaking the bank.

What Problem Does This Paper Address?

In the current "arms race" of billion-parameter language models, startups face a harsh reality: deploying humongous models is prohibitively expensive and environmentally taxing. Furthermore, general-purpose models like XLM-R often struggle with the idiosyncratic vocabulary and complex structure of legal text (contracts and regulations).

While model compression techniques like pruning and quantization exist, their combined efficacy on domain-specific, multilingual legal tasks has been understudied. This paper bridges the gap between high-level research and the practical needs of legal-tech practitioners who require efficient, accurate, and multilingual tools.

Methods

The authors followed a three-step incremental pipeline:

  1. Domain-Specific Pre-training: They built four variants of C-XLM (Tiny, Small, Base, Large) based on the RoBERTa architecture. Unlike general models, these were trained from scratch using a custom 64k-token vocabulary optimized for legal terminology across 10 languages.
  2. Fine-tuning: Models were tuned on a diverse benchmark of five legal NLP tasks.
  3. Gradual Compression (GC): This is the paper's core methodological contribution. Instead of a single compression step, they used a multi-stage approach (summarized in Algorithm 1):
  4. S0 (Teacher Assistant): Distilling the "Large" model into a "Base" model to smooth the transition.
  5. S1 (Vocabulary Pruning): Removing tokens not present in the downstream task's training set.
  6. S2 & S3 (Depth and Width Pruning): Iteratively removing layers, attention heads, and hidden units, followed by re-distillation to recover performance.
  7. S4 (Quantization): Converting weights to 8-bit integers and optimizing the computational graph via ONNX.
Figure 2: Pre-training loss curves of C-XLMs.

Datasets

  • Pre-training: A massive corpus of 1.8 billion tokens, including MultiEURLEX (EU regulations), US Code books, UK laws, and the LEDGAR dataset (US contracts). To support multilingualism, English contracts were translated into nine other languages using Neural Machine Translation.
  • Evaluation Benchmarks:
    • MultiEURLEX: Legal topic classification (multilingual).
  • UNFAIR-ToS: Detecting unfair clauses in Terms of Service (multilingual).
  • ContractNLI: Natural Language Inference for contracts.
  • Contract-Obligations: Extracting obligations from service agreements.
  • ContractNER: Named Entity Recognition for contract elements.

Key Insights and Ideas

  • Domain Specificity Trumps Scale: C-XLM models consistently outperformed generic XLM-R models of the same size. In some cases, a smaller domain-specific model was competitive with a much larger generic one.
  • The "Gradual" Advantage: The Gradual Compression pipeline proved superior to direct distillation. By taking incremental steps and using "Teacher Assistants," the models retained more "knowledge" during the shrinking process.
  • Vocabulary Pruning is Highly Effective: For specific tasks, pruning the vocabulary (Step S1) significantly reduced parameter counts (up to 31% for ContractNLI) with almost no loss in accuracy.
  • Language Disparity: Even with balanced training data, performance varied across languages (e.g., English vs. German). The authors hypothesize this is due to the inherent "idiosyncrasies" of legal language in different jurisdictions rather than data imbalance.
Figure 1: MLM performance per language across C-XLM model variants depicted with different coloured webs.

Key Results

  • Efficiency: The fully compressed (GC) models were approximately 42x smaller and 66x faster than the original "Large" teacher model.
  • Performance Retention: Despite the massive reduction in size, the average performance drop across all tasks was only 2.3 percentage points.
  • Superiority over Baselines: Compressed models produced via the GC pipeline outperformed equally-sized models that were simply fine-tuned or distilled in a single step. In the "Obligation" extraction task, the compressed model actually outperformed its own "Large" teacher.

Limitations and Future Work

  • The "No Free Lunch" Rule: The authors acknowledge that there is a limit to compression; eventually, further shrinking leads to a steep performance cliff.
  • Scaling Up: The study was limited to models in the hundreds of millions of parameters. Testing the GC pipeline on billion-parameter models (like GPT-3 or Llama variants) remains a goal for future research, resource permitting.
  • Privacy and Data Leakage: While smaller models allow for safer on-premise deployment, the paper notes that the problem of data leakage in legal-tech remains an open challenge.
  • Environmental Trade-offs: While inference is much greener, the initial pre-training from scratch is still compute-intensive. Future work could explore more efficient pre-training strategies.

Summary updated

Abstract

In the era of billion-parameter-sized Language Models (LMs), start-ups have to follow trends and adapt their technology accordingly. Nonetheless, there are open challenges since the development and deployment of large models comes with a need for high computational resources and has economical consequences. In this work, we follow the steps of the R&D group of a modern legal-tech start-up and present important insights on model development and deployment. We start from ground zero by pre-training multiple domain-specific multi-lingual LMs which are a better fit to contractual and regulatory text compared to the available alternatives (XLM-R). We present benchmark results of such models in a half-public half-private legal benchmark comprising 5 downstream tasks showing the impact of larger model size. Lastly, we examine the impact of a full-scale pipeline for model compression which includes: a) Parameter Pruning, b) Knowledge Distillation, and c) Quantization: The resulting models are much more efficient without sacrificing performance at large.
BibTeX
@misc{maroudas2022legal,
  author = {Stelios Maroudas and Sotiris Legkas and Prodromos Malakasiotis and Ilias Chalkidis},
  title = {{Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Models}},
  year = {2022},
  eprint = {2210.13086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.13086},
}
Figure