\textit{StateCensusLaws.org}: A Web Application for Consuming and Annotating Legal Discourse Learning

Alexander Spangher, Jonathan MayarXiv 2021 · arXiv

Introduction

U.S. state laws are notoriously difficult to analyze at scale. While federal statutes are relatively accessible, state-level legislation is fragmented across fifty jurisdictions, often obscured by paywalls or anti-scraping measures on platforms like LexisNexis. Beyond simple accessibility, a "functional" gap exists: standard keyword searches cannot easily answer structural questions regarding how specific population thresholds trigger legal consequences.

StateCensusLaws.org addresses this by providing an open-source platform to scrape, parse, and annotate state laws tied to U.S. Census data. By combining a custom-built corpus with a novel discourse-tagging schema and BERT-based NLP models, the platform allows researchers and journalists to "decode" how population data determines local governance, from liquor licenses to judicial appointments.

Technical Framework and Modeling

The platform utilizes a three-pronged technical approach to bridge the gap between raw legal text and actionable data:

  1. Scraping Infrastructure: The authors developed 25 Dockerized web scrapers using Scrapy and Selenium. These are designed to mimic human browsing behavior to bypass the aggressive anti-scraping mechanisms often employed by legal database providers.
  2. Modeling: The system employs a BERT-CRF (Conditional Random Field) model for sequence labeling. This model automatically identifies functional spans within raw legal text, providing a baseline for discourse parsing in the legal domain.
  3. Human-in-the-Loop Annotation: To improve the model and create a gold-standard dataset, the authors built a modular JavaScript annotation interface. This tool allows users to highlight text and draw relations between spans, creating a virtuous cycle of data improvement as journalists use the tool.

The Functional Discourse Schema

The core innovation of the project is a five-part discourse schema that focuses on the logic of the law rather than mere semantic entities. This schema identifies:

  • Subject: The entity gaining power or restriction.
  • Consequence: The specific power or restriction conferred.
  • Object: The entity affected by the subject’s action.
  • Test: The explicit condition (e.g., "population above 10,000").
  • Probe: The entity being tested (e.g., "counties").
The StateCensusLaws.org annotation interface, allowing users to tag discourse spans and define relationships between legal entities.

User Workflows and Discovery

The application supports two primary discovery paths for researchers. "Flow 1" follows a traditional keyword search path that leads directly to full-text legal documents. "Flow 2" enables a more granular exploration of the corpus by filtering for specific discourse spans—such as viewing all laws that share a specific "Consequence" or "Test" trigger.

System workflows illustrating the search-to-text path (Flow 1) and the span-exploration path (Flow 2).

Insights and Journalistic Impact

This functional approach has revealed the phenomenon of "stealth" legislation—laws written with extremely narrow population windows (e.g., applying only to counties with 335,000 to 336,000 residents) to target specific municipalities without naming them.

In pilot studies, journalists used the tool to identify 270 laws in Tennessee, New York, and Illinois where liquor license allocations were population-dependent. Their findings highlighted that mid-sized cities are the most vulnerable to the fallout of a 5-10% Census undercount, as such shifts can trigger significant changes in local regulatory authority.

Conclusion and Future Work

While StateCensusLaws.org provides a powerful new utility for legal discourse analysis, several limitations remain. The current interface lacks full accessibility features for seeing-impaired users, and the system is presently limited to English-language U.S. laws. Future development will focus on expanding the scope to other legal systems and developing models that can automatically extract the relational links between identified discourse spans.

Summary updated

Abstract

In this work, we create a web application to highlight the output of NLP models trained to parse and label discourse segments in law text. Our system is built primarily with journalists and legal interpreters in mind, and we focus on state-level law that uses U.S. Census population numbers to allocate resources and organize government. Our system exposes a corpus we collect of 6,000 state-level laws that pertain to the U.S. census, using 25 scrapers we built to crawl state law websites, which we release. We also build a novel, flexible annotation framework that can handle span-tagging and relation tagging on an arbitrary input text document and be embedded simply into any webpage. This framework allows journalists and researchers to add to our annotation database by correcting and tagging new data.
BibTeX
@misc{spangher2021textit,
  author = {Alexander Spangher and Jonathan May},
  title = {{\textbackslash\{\}textit\{StateCensusLaws.org\}: A Web Application for Consuming and Annotating Legal Discourse Learning}},
  year = {2021},
  eprint = {2104.10263},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.10263},
}
Figure