Augmenting Human-Centered Racial Covenant Detection and Georeferencing with Plug-and-Play NLP Pipelines

Jiyoon Pyo, Yuankun Jiao, Yao‐Yi Chiang, Michael Corey

Introduction

Throughout the 20th century, racial covenants—clauses embedded in property deeds that legally prohibited non-White individuals from owning or occupying land—served as a primary mechanism for institutionalizing segregation and wealth inequality in the United States. While these covenants were rendered unenforceable by the 1948 Supreme Court decision in Shelley v. Kraemer and later outlawed by the Fair Housing Act of 1968, their legacy persists in modern urban geographies.

The Mapping Prejudice project seeks to document these historical injustices by identifying and geolocating every racial covenant in specific regions. This effort relies heavily on community-led crowdsourcing, where thousands of volunteers transcribe and verify historical records. However, the project faces significant technical bottlenecks: high "noise" in document detection and the extreme difficulty of georeferencing deeds that use archaic legal land descriptions rather than modern street addresses. This paper presents a two-stage NLP framework designed to augment this human-centered workflow through improved detection and automated georeferencing.

Methodology: A Two-Stage NLP Framework

The proposed system introduces two modular, "plug-and-play" pipelines that integrate into existing crowdsourcing platforms to streamline the volunteer experience.

1. Racial Entity Identification

Traditional methods for finding covenants often rely on "fuzzy matching" keyword searches. These methods frequently flag irrelevant documents—such as those containing a person's name that happens to match a racial term—leading to volunteer fatigue. To address this, the researchers implemented a context-aware detection model using ModernBERT.

  • Token Classification: Unlike simple lexicons, the transformer-based model performs token classification to distinguish between restrictive language and benign mentions. By analyzing the surrounding context, the model identifies the intent of the clause.
  • Data Augmentation: To improve the model's sensitivity to discriminatory language, the training data was supplemented with sentences from general stereotype datasets, including StereoSet and CrowS-Pairs.
  • OCR Robustness: Historical documents are often plagued by poor scan quality. The NLP pipeline demonstrated a superior ability to "read through" errors introduced by Optical Character Recognition (OCR) compared to traditional string-matching techniques.

2. Georeferencing Module

Georeferencing historical deeds is complex because they typically reference the Public Land Survey System (PLSS)—a grid-based system of Townships, Ranges, and Sections—or the Recorded Plat Survey System (RPSS), which uses subdivisions and lot numbers.

The researchers developed a custom spaCy v3 Named Entity Recognition (NER) model to extract these specific legal entities. These extracted values are then processed by a "resolver" that matches the text against historical geographic databases. This allows the system to automatically suggest the physical location of a property, specifically targeting 1x1 mile or 6x6 mile grid blocks.

Experimental Setup and Datasets

The study utilized a massive corpus provided by the Mapping Prejudice project, encompassing over 1.4 million historical property record pages from four counties in Minnesota and Wisconsin. For the georeferencing component, the team used a curated dataset of 3,453 racially covenanted deeds and 12,180 associated parcel polygons across six counties to validate the accuracy of the spatial resolutions.

Key Results

The implementation of these NLP pipelines yielded significant improvements in both efficiency and accuracy:

  • Precision and Recall: The ModernBERT-based detection model reduced false positives by 25.96% compared to the baseline keyword-matching system. Crucially, it maintained a high recall of 91.73%, ensuring that very few actual covenants were missed.
  • Mapping Accuracy: The georeferencing pipeline achieved 85.58% accuracy at a 1x1 square-mile resolution and 86.39% at a 6x6 square-mile resolution. This significantly narrows the search area for volunteers tasked with final verification.
  • Workflow Efficiency: By filtering out nearly a quarter of the irrelevant documents previously shown to volunteers, the system allows the project to scale more effectively to new jurisdictions.

Human-Centered AI Philosophy

A core tenet of this research is that AI should augment, not replace, the human element of historical research. The authors argue that the act of transcription by volunteers is a form of social activism that raises public awareness of systemic racism. The goal of the NLP framework is to remove the "tedious" aspects of the work—filtering junk data and deciphering legal jargon—allowing volunteers to focus on the meaningful work of historical analysis and community engagement.

Limitations and Future Directions

While successful, the system faces several ongoing challenges:

  • Implicit Bias: The model occasionally struggles with "implicit" restrictive clauses that describe exclusionary practices without using specific racial slurs.
  • Subdivision Resolution: While the system excels at finding PLSS grid blocks, resolving specific subdivisions (RPSS) remains difficult due to the high variability in historical recording practices.
  • Regional Specificity: Because land survey systems vary by state, the current model is optimized for the Midwest. Future work will focus on adapting the pipeline to different historical land-grant systems used in other parts of the U.S.

Conclusion

By combining state-of-the-art transformer models with specialized NER pipelines, this research demonstrates how NLP can be used to confront the digital remains of systemic racism. The "plug-and-play" nature of these tools ensures they can be integrated into various historical mapping projects, accelerating the vital work of documenting and understanding the geography of prejudice.

Summary updated

Abstract

Though no longer legally enforceable, racial covenants in 20th-century property deeds continue to influence spatial and socioeconomic inequalities. Understanding this legacy requires identifying racially restrictive language and geolocating the associated properties. The Mapping Prejudice project addresses this by engaging volunteers using the Zooniverse crowdsourcing platform to transcribe racial covenants from scanned property deed images and using transcribed legal descriptions to link those covenants to modern parcel maps. Mapping Prejudice does not seek to fully automate racial covenants research, as it has found that crowdsourcing has powerful social benefits in addition to its technical superiority over fully automated methods. The project has historically relied on simple lexicon-based searching and, more recently, on fuzzy matching to flag suspected racial covenants for community mapmakers to review. However, the introduction of fuzzy matching has significantly increased the number of false positives that volunteers must review, negatively impacting their user experience and potentially presenting scalability challenges. Further, while many properties can be mapped automatically based on user transcription, others require time-consuming manual geolocation via human research.
BibTeX
@misc{pyo2025augmenting,
  author = {Jiyoon Pyo and Yuankun Jiao and Yao‐Yi Chiang and Michael Corey},
  title = {{Augmenting Human-Centered Racial Covenant Detection and Georeferencing with Plug-and-Play NLP Pipelines}},
  year = {2025},
  eprint = {2509.05829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.05829},
}
Figure