TL;DR
This paper introduces two plug-and-play NLP pipelines designed to assist volunteers in mapping historical racial covenants by automatically detecting discriminatory language and georeferencing property locations from deed text. By fine-tuning ModernBERT for token classification and building a custom spaCy NER model for land survey systems, the authors drastically reduce the false-positive burden on human reviewers while achieving high georeferencing accuracy.
What Problem Does This Paper Address?
Racial covenants—historical, legally binding clauses inserted into 20th-century property deeds to prohibit non-White individuals from buying or renting homes—have left a persistent legacy of urban segregation and wealth inequality. Public history initiatives like the Mapping Prejudice project rely on crowdsourced volunteers to find and map these covenants from millions of scanned deed pages.
However, the project's current keyword and fuzzy-matching search tools suffer from high false-positive rates, forcing volunteers to wade through thousands of irrelevant documents. Furthermore, translating archaic, complex legal land descriptions into modern geographic coordinates remains a labor-intensive manual task. This paper tackles these bottlenecks by introducing high-precision NLP tools designed to augment, rather than replace, the human-in-the-loop workflow.
Methods
The authors developed two modular, plug-and-play pipelines:
- Racial Entity Identification: Instead of relying on rigid lexicons or fuzzy string matching, the authors fine-tuned ModernBERT for token classification. ModernBERT was selected for its long context window and fast inference speed, which are critical for processing lengthy legal documents. The model leverages sentence-level context to identify previously unseen variants of discriminatory language and successfully recover text distorted by Optical Character Recognition (OCR) errors.
- Georeferencing Module: The authors trained a spaCy v3 Named Entity Recognition (NER) model from scratch to extract seven geographic entities spanning two historical land-survey frameworks: the Recorded Plat Survey System (RPSS: State, County, City, Subdivision) and the Public Land Survey System (PLSS: Township, Range, Section). A rule-based resolver normalizes these textual descriptions (e.g., converting "Township numbered Eight North" to numeric values) and matches them against modern GIS shapefiles.
Datasets
- Mapping Prejudice Dataset: Comprises over 1.4 million historical property deed pages from five counties in Minnesota and one in Wisconsin. Token-level training utilized 147,555 labeled sentences from 7,244 documents. Georeferencing evaluation relied on 3,453 racially covenanted deeds mapped to 12,180 manually digitized parcel polygons.
- Stereotype Corpora: To improve the text classifier's generalizability, the authors supplemented their training data with 3,362 sentences extracted from StereoSet and CrowS-Pairs.
- Public GIS Data: Official PLSS geographic datasets from the Minnesota and Wisconsin Departments of Natural Resources were used to resolve extracted text to real-world coordinates.
Key Insights and Ideas
- Human-AI Collaboration over Full Automation: The authors explicitly reject replacing humans with AI. Crowdsourcing has profound social benefits, as interacting with primary source documents educates volunteers on structural racism. The AI's role is strictly to enhance the volunteer experience by filtering out noise.
- Context Defeats OCR Noise: Traditional fuzzy matching fails when OCR heavily distorts words. A transformer-based token classifier leverages surrounding context to correctly flag a clause even if the specific racial term is misspelled or mangled during scanning.
- The Challenge of Historical "City" Entities: In historical deeds, city names are highly variable, abbreviated, or omitted entirely, making "City" the hardest entity to recognize (F1-score of only 48.06%). Conversely, "Subdivision" names are highly structured and perform exceptionally well (91.88% F1).
Key Results
- Racial Term Detection: The proposed ModernBERT model boosted token-level precision to 93.69% (a 25.96% improvement over fuzzy matching) and document-level precision to 93.06% (a 34.03% improvement), while maintaining a strong recall of 91.73%.
- Georeferencing Accuracy: The pipeline achieved 85.58% accuracy at a granular 1×1 square-mile resolution (Township, Range, Section) and 86.39% accuracy at a 6×6 square-mile resolution.
Limitations and Future Work
- Implicit Language: The model struggles with implicit or legally coded restrictive clauses (e.g., "No nationality, servants other different than race...") because it was trained primarily on explicit racial terms. The authors propose a hybrid, two-stage document-to-token classification approach for future work to capture these edge cases.
- Geographic Generalizability: Legal traditions, survey systems, and archival practices vary significantly across U.S. states. The current model is optimized for the Upper Midwest (Minnesota/Wisconsin) and would require local adaptation to scale nationally.
- Subdivision-level Granularity: Only 16.18% of deeds currently resolve down to the exact subdivision level due to the high diversity of RPSS variants, highlighting a need for better normalization and transformer-based NER for land descriptions.
Summary updated