CivicBench: A Benchmark and Memory Framework for Web-Based Civic Data Analytics

Yangyu Wu, Siru Zhong, Haoyuan Liang, Yuxuan LiangarXiv 2026 · arXiv

TL;DR

CivicBench is a new 400-task benchmark designed to test how well LLM agents handle the messy, siloed world of municipal data (like 311 reports and weather records). Alongside it, the authors introduce Task-State Memory (TSM), a structured memory framework that ensures agents don't just hallucinate numbers, but instead produce auditable claims that respect spatial and temporal boundaries.

What Problem Does This Paper Address?

Civic data analysis—like comparing service request volumes across city districts or checking if weather impacted road repairs—is a "Web data integration" nightmare. Municipalities publish data across different portals with varying schemas, update cycles, and geographic units.

While tool-using LLM agents (like those using ReAct) can fetch this data, they often suffer from behavioral state decay. They lose track of the "why" behind a calculation, mix up data from different years or cities, or fail to realize when the available data simply doesn't support the requested conclusion. Existing benchmarks usually score agents on whether they finish a task, but they rarely check if the evidence chain remains intact and correctly scoped.

Methods

The authors tackle this with a two-pronged approach:

  1. CivicBench Benchmark: A suite of 400 tasks that requires agents to integrate data from six public sources. Unlike standard benchmarks, it uses "executable contracts" to score agents on computation, provenance (source linkage), coverage, and scope. It specifically tests for scope-aware reasoning: the ability to provide a "Qualified" answer (e.g., "This only applies to District A") or a grounded "Refusal" when data is missing.

    Figure 2: Benchmark construction pipeline. Evidence-derived targets and contracts are fixed before governance-facing questions are framed. Public tasks, evaluator ground truth, and internal generation state remain separate, while target/scorer, operation, and no-record shortcut checks gate the frozen artifact.
  2. Task-State Memory (TSM): Instead of relying on a messy, free-form chat transcript, TSM implements a six-stage deterministic loop:

  3. INIT: Compiles task constraints.
  4. WRITE: Converts tool observations into "typed assertions" (storing file, field, and scope).
  5. RESOLVE: Reconciles new data with old, identifying conflicts or superseding stale facts.
  6. CHECK: Evaluates the ledger against task requirements.
  7. RETRIEVE: Projects a "bounded" version of the memory back to the LLM, highlighting unresolved constraints and recent errors.
  8. DECIDE: The LLM chooses whether to continue, answer, qualify, or refuse.

    Figure 4: Task-State Memory loop and ledger structure.

Datasets

The authors introduced CivicBench, which materializes civic Web data into "frozen bundles" to ensure reproducible evaluation.

  • Scope: 4 cities (Austin, Boston, Chicago, San Francisco) over 2 years (2024–2025).
  • Sources: 311 service records (~2.5 million rows), NOAA weather observations, GeoJSON boundaries, and ACS/TIGER census data.
  • Task Composition: 400 unique questions spanning 10 capabilities, including priority ranking, trend change analysis, and data fitness assessment.
Figure 1: Benchmark scope and task composition. Six public evidence sources cover four cities and two years; each of the 400 tasks combines a computation with provenance, scope, and a claim boundary.

Key Insights and Ideas

  • The "Evidence Ledger" vs. Raw Context: The core "aha" moment is that LLMs perform better when their memory is managed as a structured database of assertions rather than a raw text log. By explicitly tracking provenance and scope, the system can catch errors (like using 2024 data for a 2025 query) before the LLM even sees it.
  • Tri-State Outputs: The benchmark moves beyond binary right/wrong scoring. By including "Qualified" and "Refusal" states, it forces agents to respect the limits of their evidence—a critical requirement for high-stakes policy work.
  • Retrieval is King: Ablation studies showed that the RETRIEVE stage—which filters the massive evidence ledger into a compact, constraint-centered projection for the LLM—is the single most important part of the TSM framework.

Key Results

The authors tested TSM against standard ReAct agents across nine different model pairs (including DeepSeek, Grok, and GPT-4 variants).

  • Performance Boost: TSM improved mean task scores for 8 out of 9 model pairs. For example, DeepSeek V4 Flash saw a score increase of +0.0669, and Grok 4 Fast Reasoning improved by +0.0600.
  • Reduced Failures: TSM significantly reduced computation failures (e.g., from 277 to 262 for DeepSeek).
  • Efficiency: Interestingly, TSM often reduced total token usage (by ~11% for DeepSeek) because the structured memory projection is more concise than a full interaction history.

Limitations and Future Work

  • Geographic Breadth: The benchmark is currently limited to four U.S. cities. The authors note that generalization claims are confined to this specific artifact.
  • Prose Quality: The scorer focuses on structured data and deterministic completeness; it does not evaluate the stylistic quality or "policy usefulness" of the agent's written analysis.
  • Legal Hurdles: Redistribution rights for raw data from cities like Chicago and Boston remain a challenge, requiring the release to separate metadata from restricted source-derived artifacts.
  • Future Directions: The authors suggest expanding to causal decomposition (understanding why a trend happened, not just that it happened) and broader validity studies.

Summary updated

Abstract

Municipalities publish service requests, geographic boundaries, weather observations, demographic estimates, and operational guidance through independently maintained Web portals. Civic analysis must combine these sources while preserving their schemas, source links, spatial and temporal scopes, and interpretive limits. Tool-using language agents can perform this integration, but existing task benchmarks rarely test whether a computed claim remains supported by the records used. We present CivicBench, a 400-task benchmark over four cities and two years that materializes civic Web data into frozen bundles for reproducible evaluation. Executable contracts score computation, source linkage, coverage, scope, qualified answers, and refusal when the requested conclusion is not supported. We also introduce Task-State Memory (TSM), which compiles tool observations into typed assertions, reconciles records only within compatible scopes, and retrieves a bounded state organized around unresolved constraints. Across nine completed model pairs, eight yield positive mean-score differences with 95\% semantic-contract-cluster bootstrap intervals excluding zero; the ninth interval crosses zero. The benchmark and memory framework make the records behind a claim inspectable at the point of decision.
BibTeX
@misc{wu2026civicbench,
  author = {Yangyu Wu and Siru Zhong and Haoyuan Liang and Yuxuan Liang},
  title = {{CivicBench: A Benchmark and Memory Framework for Web-Based Civic Data Analytics}},
  year = {2026},
  eprint = {2611.01145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01145},
}
Figure