TL;DR
This paper tackles the challenge of accurately extracting lawyer names from historical legal documents, which are often riddled with OCR errors and complex formatting. They introduce an ensemble model combining a transformer neural network with a finite state machine, achieving a ~10% accuracy improvement over existing methods and demonstrating a significant reduction in errors over time.
What Problem Does This Paper Address?
Digitizing historical legal texts, often done through Optical Character Recognition (OCR), introduces numerous errors. While standard spell checkers can fix many issues, extracting and correcting proper names, especially those of lawyers, is a persistent challenge. This unreliability hinders research into legal history, lawyer biases, and connections that might influence case outcomes. The "party names" section in legal documents, typically found at the beginning, presents a unique challenge due to its complex sentence structure and historical variations in formatting, making automated parsing difficult.
Methods
The authors propose a two-pronged approach:
- Ensemble Model for Name Extraction: They developed an ensemble model that combines a transformer-based neural network with a custom finite state machine.
- Transformer (Flair's FLERT): This component is used for Part-of-Speech (PoS) tagging, leveraging pre-trained models.
- Finite State Machine (FSM): This custom FSM takes the PoS-tagged tokens and uses pre-established knowledge of legal text syntax and style to identify potential names. This is a key differentiator, as it incorporates domain-specific rules.
- Heuristic Textual Analysis for Correction: After extraction, identified names are subjected to heuristic rules to identify and correct transcription errors, misspellings, and name variations. Techniques like the Double Metaphone algorithm are used to handle phonetic similarities and potential misspellings.
They also developed a baseline model, HCL-NER, which is an identical architecture to FLERT but trained on their specific legal dataset, serving as a comparison point.
Datasets
- Harvard Caselaw Access Project: This project provided the primary dataset, containing the full text of US court decisions dating back to the 1700s. For their analysis, they focused on cases from 1900 to 2010 from states that existed in 1900.
- CONLL03: Used for training the baseline FLERT model. This dataset consists of Reuters news stories from 1996-1997 and differs in style from legal texts.
Key Insights and Ideas
- Domain-Specific Knowledge is Crucial: Simply applying general-purpose NER models (like FLERT trained on news text) to legal documents is insufficient. The unique structure and historical variations in legal texts require specialized approaches.
- Ensemble Power: Combining a powerful neural network (transformer) with a rule-based system (FSM) that encodes domain knowledge significantly boosts performance. The FSM helps overcome limitations of purely data-driven models in specific contexts.
- OCR Errors Persist: Even with advanced NLP, OCR-introduced errors remain a significant hurdle, particularly for proper nouns.
- Error Rates Decline Over Time: The study reveals a clear trend of decreasing potential misspellings and initial usage in lawyer names from the early 20th century to the early 21st century, likely due to the shift to digital creation and storage.
- Double Metaphone for Phonetic Matching: This algorithm proves effective in grouping potentially misspelled or variably spelled names that sound alike, aiding in disambiguation.
Key Results
- Accuracy Improvement: The proposed ensemble model achieved an accuracy of 78%, outperforming FLERT (68%) and their custom-trained HCL-NER model (73%). This represents an approximate 10% improvement.
- Reduced Errors: The ensemble model significantly reduced both false positives (incorrectly identified names) and false negatives (missed names) compared to the baseline models. Specifically, it had fewer false negatives (89 vs. 159 for FLERT) and false positives (49 vs. 81 for FLERT).
- Declining Misspellings: Analysis showed that the percentage of potential misspellings in names has declined over time, with some jurisdictions showing error rates exceeding 15% before 1950, dropping significantly thereafter.
- Decreased Initial Usage: The use of initials for lawyer names has drastically decreased from 79% in Massachusetts in 1900 to around 0.1% in 2010, with Kansas showing the highest current usage at 0.7%.
Limitations and Future Work
- Focus on Lawyer Names: While the framework for identifying firms and roles is included, the current testing and evaluation primarily focus on extracting lawyer names.
- HCL-NER Dataset Size: The HCL-NER model was trained on a relatively small subset of the Harvard Caselaw Access Project. The authors suggest that using a larger dataset might improve its performance.
- Ambiguous Initials: Quantifying the exact extent of the problem with ambiguous initials is difficult without a comprehensive list of all lawyer names.
- Future Work: The authors suggest extending the state machine to identify the person's role and other data included in the text, and potentially using a larger dataset for training models like HCL-NER.
Summary updated