TL;DR
CivicBench is a new 400-task benchmark designed to test how well LLM agents handle the messy, siloed world of municipal data (like 311 reports and weather records). Alongside it, the authors introduce Task-State Memory (TSM), a structured memory framework that ensures agents don't just hallucinate numbers, but instead produce auditable claims that respect spatial and temporal boundaries.
What Problem Does This Paper Address?
Civic data analysis—like comparing service request volumes across city districts or checking if weather impacted road repairs—is a "Web data integration" nightmare. Municipalities publish data across different portals with varying schemas, update cycles, and geographic units.
While tool-using LLM agents (like those using ReAct) can fetch this data, they often suffer from behavioral state decay. They lose track of the "why" behind a calculation, mix up data from different years or cities, or fail to realize when the available data simply doesn't support the requested conclusion. Existing benchmarks usually score agents on whether they finish a task, but they rarely check if the evidence chain remains intact and correctly scoped.
Methods
The authors tackle this with a two-pronged approach:
-
CivicBench Benchmark: A suite of 400 tasks that requires agents to integrate data from six public sources. Unlike standard benchmarks, it uses "executable contracts" to score agents on computation, provenance (source linkage), coverage, and scope. It specifically tests for scope-aware reasoning: the ability to provide a "Qualified" answer (e.g., "This only applies to District A") or a grounded "Refusal" when data is missing.
Figure 2: Benchmark construction pipeline. Evidence-derived targets and contracts are fixed before governance-facing questions are framed. Public tasks, evaluator ground truth, and internal generation state remain separate, while target/scorer, operation, and no-record shortcut checks gate the frozen artifact. -
Task-State Memory (TSM): Instead of relying on a messy, free-form chat transcript, TSM implements a six-stage deterministic loop:
- INIT: Compiles task constraints.
- WRITE: Converts tool observations into "typed assertions" (storing file, field, and scope).
- RESOLVE: Reconciles new data with old, identifying conflicts or superseding stale facts.
- CHECK: Evaluates the ledger against task requirements.
- RETRIEVE: Projects a "bounded" version of the memory back to the LLM, highlighting unresolved constraints and recent errors.
-
DECIDE: The LLM chooses whether to continue, answer, qualify, or refuse.
Figure 4: Task-State Memory loop and ledger structure.
Datasets
The authors introduced CivicBench, which materializes civic Web data into "frozen bundles" to ensure reproducible evaluation.
- Scope: 4 cities (Austin, Boston, Chicago, San Francisco) over 2 years (2024–2025).
- Sources: 311 service records (~2.5 million rows), NOAA weather observations, GeoJSON boundaries, and ACS/TIGER census data.
- Task Composition: 400 unique questions spanning 10 capabilities, including priority ranking, trend change analysis, and data fitness assessment.
Key Insights and Ideas
- The "Evidence Ledger" vs. Raw Context: The core "aha" moment is that LLMs perform better when their memory is managed as a structured database of assertions rather than a raw text log. By explicitly tracking provenance and scope, the system can catch errors (like using 2024 data for a 2025 query) before the LLM even sees it.
- Tri-State Outputs: The benchmark moves beyond binary right/wrong scoring. By including "Qualified" and "Refusal" states, it forces agents to respect the limits of their evidence—a critical requirement for high-stakes policy work.
- Retrieval is King: Ablation studies showed that the RETRIEVE stage—which filters the massive evidence ledger into a compact, constraint-centered projection for the LLM—is the single most important part of the TSM framework.
Key Results
The authors tested TSM against standard ReAct agents across nine different model pairs (including DeepSeek, Grok, and GPT-4 variants).
- Performance Boost: TSM improved mean task scores for 8 out of 9 model pairs. For example, DeepSeek V4 Flash saw a score increase of +0.0669, and Grok 4 Fast Reasoning improved by +0.0600.
- Reduced Failures: TSM significantly reduced computation failures (e.g., from 277 to 262 for DeepSeek).
- Efficiency: Interestingly, TSM often reduced total token usage (by ~11% for DeepSeek) because the structured memory projection is more concise than a full interaction history.
Limitations and Future Work
- Geographic Breadth: The benchmark is currently limited to four U.S. cities. The authors note that generalization claims are confined to this specific artifact.
- Prose Quality: The scorer focuses on structured data and deterministic completeness; it does not evaluate the stylistic quality or "policy usefulness" of the agent's written analysis.
- Legal Hurdles: Redistribution rights for raw data from cities like Chicago and Boston remain a challenge, requiring the release to separate metadata from restricted source-derived artifacts.
- Future Directions: The authors suggest expanding to causal decomposition (understanding why a trend happened, not just that it happened) and broader validity studies.
Summary updated