Abstract
The application of Natural Language Processing (NLP) to specialized domains, such as the law, has recently received a surge of interest. As many legal services rely on processing and analyzing large collections of documents, automating such tasks with NLP tools such as language models emerges as a key challenge since legal documents may contain specialized vocabulary from other domains, such as medical terminology in personal injury text. However, most language models are general-purpose models, which either have limited reasoning capabilities on highly specialized legal terminology and syntax, such as BERT or ROBERTA, or are expensive to run and tune, such as GPT-3.5 and Claude. Thus, in this paper, we propose a specialized language model for personal injury text, LEGALRELECTRA, which is trained on mixed-domain legal and medical corpora. We show that as a small language model, our model improves over general-domain and single-domain medical and legal language models when processing mixed-domain (personal injury) text. Our training architecture implements the ELECTRA framework but utilizes REFORMER instead of BERT for its generator and discriminator. We show that this improves the model's performance on processing long passages and results in better long-range text comprehension.
Introduction
The legal domain presents unique challenges for Natural Language Processing (NLP). Specifically, personal injury litigation requires the synthesis of complex legal procedures with intricate medical terminology. Existing general-purpose models often fail to capture the nuances of this domain overlap. Furthermore, legal documents are characterized by their extreme length, frequently exceeding the 512-token limit of standard transformer architectures like BERT. Finally, the high cost and privacy concerns associated with large-scale proprietary models necessitate the development of efficient, specialized, and locally deployable "small" language models.
Methodology
To address these challenges, we introduce LEGALRELECTRA, a model that combines the sample-efficient ELECTRA training framework with the long-context capabilities of the Reformer architecture.
Architecture: RELECTRA
Unlike standard Masked Language Modeling (MLM), our approach utilizes the ELECTRA framework, which consists of a generator and a discriminator. The generator replaces tokens with plausible alternatives, while the discriminator learns to distinguish between original and replaced tokens. We replace the standard BERT backbone with Reformer, which utilizes Locality-Sensitive Hashing (LSH) attention. This modification reduces computational complexity from $O(L^2)$ to $O(L \log L)$, enabling a context window of 8,092 tokens—a 16-fold increase over BERT.
Custom Tokenization
A critical component of our methodology is the implementation of a domain-specific tokenizer. Standard tokenizers often fragment specialized legal and medical terms into meaningless sub-units. By training a custom tokenizer on our mixed-corpus, we ensure that specialized vocabulary is preserved as distinct semantic units.
Experimental Setup
LEGALRELECTRA was pre-trained on a 12GB mixed-domain corpus comprising:
- Legal (6GB): Case descriptions from CourtListener.
- Medical (3GB): Clinical notes from MIMIC and MIMIC-CXR.
- Mixed (3GB): Curated personal injury case descriptions and Supreme Court opinions.
Evaluation was conducted on a novel dataset of 500 personal injury cases from Kentucky and Louisiana (1998–2018) focused on Legal Case Retrieval and Named Entity Recognition (NER).
Results and Discussion
Our evaluation demonstrates that LEGALRELECTRA consistently outperforms general-domain and single-domain baselines.
Named Entity Recognition (NER)
LEGALRELECTRA achieved an F1 score of 85.93 on legal text and 78.57 on mixed-domain text. It outperformed BERT, LEGAL-BERT, and CLINICAL-BERT, particularly in identifying entities that span both domains.
Legal Case Retrieval
In zero-shot retrieval tasks, the model achieved 90% accuracy in identifying claim types and 92% accuracy for injury categories. The expanded context window proved vital; the model successfully identified parties mentioned at opposite ends of long documents, which 512-token models failed to capture.
Efficiency and Domain Synergy
The results suggest a synergistic effect: training on a combined legal-medical corpus allows the model to understand how medical jargon functions within a litigation context better than a model trained on either domain in isolation. Furthermore, the ELECTRA/Reformer combination allowed for high performance on a single NVIDIA P100 GPU.
Conclusion
LEGALRELECTRA demonstrates that specialized, long-context models can effectively bridge the gap between complex domains like law and medicine. While the model currently struggles with numerical reasoning (e.g., medical expenses), future work will focus on integrating numerical embeddings and refining datasets through expert human-in-the-loop annotation to reduce noise.
Summary updated