Inferring missing metadata from environmental policy texts

Steven Bethard, Egoitz Laparra, Sophia Wang, Yiyun Zhao, Ragheb Al-Ghezi, Aaron Lien, Laura López-HoffmanLaTeCH @ NAACL 2019 · DOI

Abstract

This paper presents a framework for extracting metadata and analyzing text reuse within the massive, unstructured archive of National Environmental Policy Act (NEPA) documents. Using a corpus of over 40,000 PDFs, we evaluate rule-based methods for agency identification and version alignment. Our results indicate that while agency identification is relatively successful (86% accuracy), version alignment is hindered by metadata inconsistencies. Furthermore, we find that over 90% of text in final Environmental Impact Statements is typically reused from draft versions, raising questions about the impact of public participation on final policy outcomes.

Introduction

The National Environmental Policy Act (NEPA) is the cornerstone of U.S. environmental policy, requiring federal agencies to conduct detailed scientific analyses—known as Environmental Impact Statements (EIS)—for any major project, from highways to oil drilling. Since 1970, over 37,000 of these statements have been produced. However, there is no central repository, and documents are scattered across various agency websites, often lacking basic metadata. This study addresses the technical hurdles of automatically extracting metadata to enable large-scale research into federal environmental decision-making.

Methodology

The researchers established baseline performances for three core challenges using rule-based approaches:

  1. Identifying Lead Agencies: A system scans the first 15 pages of a document for agency names. To handle government hierarchies, the model filters out parent organizations if a more specific sub-agency (e.g., the Forest Service vs. the USDA) is mentioned.
  2. Aligning Document Versions: To link a "Draft EIS" to its "Final EIS," three matching strategies were tested: exact title matches, metadata matches (Date + Agency + State), and a hybrid approach.
  3. Detecting Reused Text: Using a baseline from the PAN Plagiarism Detection task, texts were partitioned into 50-character chunks to calculate a "DRAFT-REUSE" score—the fraction of text in a final version copied directly from the draft.

Data Collection

The researchers compiled a massive corpus by web-crawling government sites, collecting over 40,000 NEPA-related documents from the EPA, DOI, and DOE. For evaluation, they used a manually curated dataset from NEPA.gov containing metadata for 1,161 EISs published between 2010 and 2017 as a "ground truth."

Results and Discussion

Version Alignment Performance

While the rule-based matching models were highly precise, they suffered from low recall. The best-performing model, which combined title and metadata matching, only achieved a recall of 67.4%.

Precision and recall metrics for different EIS version alignment strategies.

This "metadata gap" highlights the inconsistency of government record-keeping; even minor typographical errors in a title can break automated alignment systems.

Text Reuse and the "90% Rule"

One of the most striking findings is the high degree of textual similarity between draft and final versions of the same project.

Histogram of text reuse scores showing a heavy skew toward the 0.9–1.0 range.

As shown in the histogram, the majority of final EIS documents reuse 90% or more of the text from their draft versions. This suggests that public comments—intended to influence the final outcome—often result in only marginal changes to the document text.

Conclusion and Future Work

The study identifies several bottlenecks for future research, including the difficulty of PDF-to-text conversion for complex layouts and the labor-intensive nature of scraping non-standardized government websites. Future work will explore machine learning classifiers, such as those using the Universal Sentence Encoder, to handle more nuanced contexts where agencies are mentioned but are not the primary authors. The team is currently hosting their findings and tools at nepaccess.org to facilitate further exploration of this environmental data.

Summary updated

BibTeX
@inproceedings{bethard2019inferring,
  author = {Steven Bethard and Egoitz Laparra and Sophia Wang and Yiyun Zhao and Ragheb Al-Ghezi and Aaron Lien and Laura López-Hoffman},
  title = {{Inferring missing metadata from environmental policy texts}},
  booktitle = {LaTeCH @ NAACL},
  year = {2019},
  doi = {10.18653/v1/w19-2506},
}
Figure