Not Your Typical Government Tipline: LLM-Assisted Routing of Environmental Protection Agency Citizen Tips

Sharanya Majumder, Zehua Li, Derek Ouyang, Kit T. Rodolfa, Elena Eneva, Julian Nyarko, Daniel E. Ho2025 · DOI

Abstract: Regulatory agencies often operate with limited resources and rely on tips from the public to identify potential violations. However, processing these tips at scale presents significant operational challenges, as agencies must correctly identify and route relevant tips to the appropriate enforcement divisions. Through a case study, we demonstrate how advances in large language models can be utilized to support overburdened agencies with limited capacities. In partnership with the U.S. Environmental Protection Agency, we leverage previously unstudied citizen tips data from their "Report a Violation" system to develop an LLM-assisted pipeline for tip routing. Our approach filters out 80.5% of irrelevant tips and increases overall routing accuracy from 31.8% to 82.4% compared to the current routing system. At a time of increased focus on government efficiencies, our approach provides a constructive path forward by using technology to empower civil servants.

Introduction: The Bottleneck of Public Oversight

Regulatory agencies like the U.S. Environmental Protection Agency (EPA) rely heavily on public tips to identify environmental violations. However, the sheer volume of submissions creates a massive administrative bottleneck. Currently, the EPA utilizes a heuristic system—essentially a set of rigid "if-then" rules based on checkboxes filled out by the submitter—to route tips to either the Civil or Criminal Division.

This system is notoriously inefficient. It fails to filter out tips that fall outside the EPA’s jurisdiction (such as noise complaints) and frequently misroutes civil matters to the Criminal Division. This results in "participatory fatigue" for citizens and a significant waste of resources for desk officers, who spend hours manually re-routing misclassified tips. The Criminal Division, in particular, has historically seen a precision of only 22.5%, meaning nearly 78% of their assigned tips were irrelevant or belonged elsewhere.

Diagram of the EPA's internal tip routing process showing the complex manual desk review steps.

Methodology: A Two-Stage LLM Pipeline

To address these inefficiencies, we propose a two-stage pipeline using Mistral-7B fine-tuned with LoRA (Low-Rank Adaptation). Unlike the current system that relies on structured form fields, this pipeline analyzes the raw free-text descriptions provided by citizens. The core insight is that submitters are often poor judges of their own tips' legal category; while they might check "intentional violation," their text description might reveal a simple civil spill. The LLM can extract nuances like "culpability" and "environmental impact" that structured fields miss.

The pipeline consists of two primary components:

  1. Relevance Filter: A binary classifier that determines if a tip is within the EPA's scope or irrelevant.
  2. Triage Model: A second classifier that routes relevant tips to either the Civil Division (for standard regulatory issues) or the Criminal Division (for intentional, high-impact crimes).

Data and Experimental Setup

The study utilized a unique, previously unstudied dataset of 19,109 tips provided by the EPA, spanning 2018 to 2024. The data included 1,371 irrelevant tips (e.g., neighbor disputes, conspiracy theories), 12,958 Civil Division tips, and 4,780 Criminal Division tips. We compared Mistral-7B against several baselines, including traditional statistical methods (TF-IDF+SVM) and other transformers (DistilBERT, RoBERTa, Llama-3.1-8B, and Qwen-2.5-7B). Mistral-7B offered the best balance of performance and probability calibration, which is vital for setting operational thresholds.

Results: Efficiency Gains and Noise Reduction

The LLM-assisted approach outperformed the heuristic system across every major metric. Overall accuracy jumped from 31.8% to 82.4%. The relevance filter successfully discarded 80.5% of irrelevant tips before they ever reached a human desk officer, significantly reducing the "noise" in the system.

Sankey diagram comparing the flow of tips in the LLM-assisted approach versus the heuristic approach, highlighting the massive improvement in precision.

Furthermore, precision for the Criminal Division rose from 22.5% to 70.5%, reducing their total workload by 74.2%. The number of civil matters incorrectly sent to the Criminal Division—which previously required manual re-routing—dropped by 84.4%.

Discussion: Balancing Recall and Precision

While precision skyrocketed, the LLM's recall for criminal matters (79.2%) was lower than the heuristic system (98.5%). This means approximately 20% of criminal leads would initially be sent to the Civil desk, causing a slight delay in response. However, in government routing, the costs are asymmetric; the authors used F1-optimized thresholds that can be tuned. By lowering the triage threshold, the EPA can prioritize high recall for criminal matters at the expense of slightly more manual review.

Future work could integrate structured fields (like location or facility names) with text to link tips to existing EPA enforcement databases. While fine-tuned LLMs remain "black boxes," tools like feature attribution help confirm the model is picking up on legally relevant signals—like "intentionally altering" for criminal leads—rather than spurious correlations. This approach provides a constructive path forward by using technology to empower civil servants and improve environmental oversight.

Summary updated

Abstract

Regulatory agencies often operate with limited resources and rely on tips from the public to identify potential violations.However, processing these tips at scale presents significant operational challenges, as agencies must correctly identify and route relevant tips to the appropriate enforcement divisions.Through a case study, we demonstrate how advances in large language models can be utilized to support overburdened agencies with limited capacities.In partnership with the U.S. Environmental Protection Agency, we leverage previously unstudied citizen tips data from their "Report a Violation" system to develop an LLM-assisted pipeline for tip routing.Our approach filters out 80.5% of irrelevant tips and increases overall routing accuracy from 31.8% to 82.4% compared to the current routing system.At a time of increased focus on government efficiencies, our approach provides a constructive path forward by using technology to empower civil servants.
BibTeX
@misc{majumder2025not,
  author = {Sharanya Majumder and Zehua Li and Derek Ouyang and Kit T. Rodolfa and Elena Eneva and Julian Nyarko and Daniel E. Ho},
  title = {{Not Your Typical Government Tipline: LLM-Assisted Routing of Environmental Protection Agency Citizen Tips}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1788},
}
Figure