Abstract
Legal and medical domains present unique challenges for natural language processing, particularly in personal injury law where documents are exceptionally long and vocabularies are highly specialized. This paper introduces LegalRelectra, a specialized language model that integrates the Reformer architecture into an ELECTRA training framework. By extending the context window to 8,192 tokens and utilizing a custom mixed-domain tokenizer, LegalRelectra significantly outperforms existing models like Legal-BERT and Clinical-BERT on domain-specific Named Entity Recognition (NER) tasks.
Introduction
Most state-of-the-art language models, such as BERT, suffer from two primary limitations when applied to the legal field. First, the "Short-Sighted Window" restricts models to 512 tokens, whereas legal documents often span thousands of words, requiring long-range context to identify key entities or rulings. Second, "Domain Isolation" prevents models from effectively handling hybrid texts; Legal-BERT excels in law but struggles with medicine, while Clinical-BERT does the opposite. In personal injury cases, where legal proceedings frequently discuss complex medical diagnoses, this isolation is a significant hurdle.
Furthermore, standard tokenizers often fragment specialized terminology into meaningless sub-words (e.g., "gastrointestinal" becoming five or six pieces), which degrades the model's ability to learn technical semantics.
Methodology
To address these challenges, the researchers developed RELECTRA, a novel architecture combining two efficient paradigms:
- The ELECTRA Framework: Unlike standard BERT models that predict masked words, ELECTRA utilizes a "Generator" to replace tokens and a "Discriminator" to identify which tokens were swapped. This approach is significantly more sample-efficient for pre-training.
- The Reformer Backbone: To overcome the quadratic complexity of standard attention mechanisms, the authors replaced the BERT backbone with the Reformer model. By using Locality-Sensitive Hashing (LSH), the computational complexity is reduced from $O(L^2)$ to $O(L \log L)$, enabling a context window of 8,192 tokens—a 16x increase over standard BERT.
Crucially, the model employs a custom Byte-Pair Encoding (BPE) tokenizer trained specifically on a mixed legal-medical corpus. This ensures that technical terms like "orthopnea" or "adjudications" remain intact as single units, acting as a force multiplier for model accuracy.
Datasets and Evaluation
LegalRelectra was pre-trained on a 12GB corpus categorized into three segments:
- Legal (6GB): Excerpts from US case law.
- Medical (3GB): Clinical notes and letters from the MIMIC and MIMIC-CXR databases.
- Mixed (3GB): Personal injury case descriptions and academic literature sourced from platforms like CourtListener.
Evaluation focused on Named Entity Recognition (NER) across general (CoNLL 2003), medical (MIMIC III), and specialized in-house legal/mixed-domain datasets.
Key Results and Insights
LegalRelectra consistently outperformed its peers on domain-specific tasks:
- Legal NER: The model achieved an F1 score of 85.93, surpassing Legal-BERT (79.32) and standard BERT (80.45).
- Mixed-Domain NER: On personal injury texts, it reached an F1 of 78.57, outperforming both Clinical-BERT (78.55) and Legal-BERT (77.07).
- Efficiency: Despite its performance, the model was trained on a single NVIDIA P100 GPU for approximately 16 days, proving that high-performance domain models are accessible without massive industrial compute clusters.
The results underscore that for specialized fields like personal injury law, "cross-training" across distinct domains is more effective than fine-tuning a single-domain model. Additionally, the 8k token window allows the model to maintain a "global" understanding of a case, which is essential when entities are mentioned once and referred to obliquely throughout a long document.
Limitations and Future Work
While promising, the research faces hurdles regarding data scarcity; "pure" personal injury data is often private, necessitating the use of supplemental general texts. Furthermore, the legal NER training data relied partly on automated string matching, which is less precise than manual expert annotation. Future research will explore LegalRelectra’s performance on more complex tasks, such as automated legal summarization and judgment prediction.
Summary updated