Read and Ignore: What Self-Directed LLM Agents Do in Public Service Scenarios

Diana Kozachek, Kinjalk SrivastavaarXiv 2026 · arXiv

TL;DR

This paper introduces a new diagnostic benchmark to evaluate how LLM agents behave when given "underspecified" instructions (e.g., "see where we stand") in realistic public-service scenarios. The researchers found that while agents are diligent readers, they rarely take initiative to act unless given an explicit task—and when they do act, they often skip the reading or produce reports that contradict the source records.

What Problem Does This Paper Address?

In public service sectors like healthcare and governance, staff often give vague, open-ended instructions to assistants. As we move toward deploying LLM agents in these roles, a critical question arises: what does an agent do with its "discretion"?

Existing benchmarks generally fall into two camps:

  1. Task-oriented: Measuring if an agent can complete a highly specific goal (like fixing a GitHub issue).
  2. Empty Sandboxes: Observing agent behavior in "context-free" environments where there is nothing to actually do.

Neither captures the reality of a workplace where an agent has access to a rich file tree of records, stakeholders, and procedures but no clear "to-do" list. This paper addresses the lack of evaluation for self-directed behavior—the ability of an agent to identify what needs doing and execute it accurately without being micro-managed.

Methods

The authors developed four expert-validated scenarios representing recurring operational cycles:

  • Governance: Participatory budgeting.
  • Healthcare: Ward planning.
  • Education: Course catalog assembly.
  • Environmentalism: Waterway monitoring.

Each scenario is a static file tree containing "decision levers"—intentional gaps, inconsistencies, or equity shortfalls that an intelligent agent should notice.

The study collected 2,400 trajectories from ten different LLMs (ranging from open-weight models like Llama-3.1-8B to frontier APIs like GPT-4o-mini) under four prompt conditions, ranging from total "freedom" to an explicit "task baseline." To classify behavior, they used a rule-based taxonomy of 9 labels and 22 subcategories, allowing for automated, reproducible analysis of nearly 94,000 individual turns.

Datasets

  • Public Service Scenarios: Four new, expert-validated directory trees with synthetic records, procedures, and timelines.
  • Trajectory Dataset: 2,400 runs (scenario-based) plus 600 directed runs.
  • Context-Free Baseline: 1,000 runs in an empty sandbox from prior work used for comparative analysis.

Key Insights and Ideas

  • The Discretion Paradox: When given freedom, agents read almost everything (96.9% of runs) but act on almost nothing (only 18.7% produce an artifact). Conversely, when given an explicit task, they act (86%) but often stop reading the context (only 50% read the domain files).
  • Context as a Stabilizer: Adding domain context (files and records) reduces "boundary probing"—the tendency of agents to test the limits of their environment or try to "escape" the sandbox—by 60%.
  • Model "Personalities": Models differ more from each other than they do across different domains. As seen in Figure 1, runs of the same model tend to cluster together in the output space, showing consistent "behavioral signatures" like aimless wandering, task looping, or deliberate auditing.
Figure 1: Output space of all 2,400 scenario runs (TF-IDF and truncated SVD of each run’s concate- nated outputs), colored by model, with six annotated runs: verbatim task execution, task looping, deliberate completion, aimless wandering, work in the domain files, and a self-invented analysis agenda. Runs of one model tend to lie close together: the models differ more than the prompt conditions.
  • The Groundedness Gap: Even when agents produce reports, they are not always reliable. 1.4% of open-ended runs produced artifacts that directly contradicted the records (e.g., citing wrong budget figures). For some frontier models, this "contradiction rate" was as high as 35% in directed tasks.

Key Results

The researchers found that a model's tendency to produce work is surprisingly consistent across environments. The ranking of which models were most "productive" in a context-free sandbox correlated strongly (Spearman 0.79) with their productivity in complex scenarios.

Table 2: Share of open-ended runs that produce an artifact or a self-chosen task, with and without domain context, at a common 50-turn cap. Brackets are 95% bootstrap intervals (10,000 resamples). Scenario n=180 per model (four domains), context-free n=75. †Gemma’s context-free runs are dominated by parse failures (54.7% of turns), so its context-free row is not comparable; every contrast is reported with and without it.

However, the reading behavior did not carry over (correlation of only 0.19). A model that reads a lot in an empty sandbox might ignore the records entirely when placed in a real-world scenario.

Quantitatively, "Environment Exploration" dominated the agents' time, accounting for 84.1% of all turns, while actual "Artifact Creation" was a mere 3.7%.

Table 1: Turn-level behavior distribution over all 93,594 turns (rule-based taxonomy), percentages of all turns. Label names state what the rule detects: code execution fires on run_python and does not indicate verified self-improvement, and termination / refusal pools deliberate completion with refusal. Each rule’s trigger is documented next to its metric definition in the release.

Limitations and Future Work

  • Static Environments: The scenarios are "frozen" in time. In a real office, files change, colleagues message you, and deadlines shift. The authors acknowledge that agents might look more active in a dynamic environment.
  • No Social Interaction: The current action set does not allow agents to ask a human for clarification, which is a standard behavior for a real-world assistant.
  • Rule-Based Coarseness: While the rule-based taxonomy is highly reproducible, it is less nuanced than human expert judgment, particularly regarding the quality or substance of the work produced.
  • Future Directions: The authors suggest this benchmark should be used as a pre-procurement "audit" for public institutions to test how specific models handle their own internal records before deployment.

Summary updated

Abstract

Underspecified instructions leave LLM agents to decide what to do. In public services, staff write these instructions, but the agent's decisions affect residents, patients, and students. We build four expert-validated public-service scenarios with records and decision levers that can be checked in the files, collect $2{,}400$ trajectories from ten LLMs under four prompt conditions, and compare them with $1{,}000$ runs in a context-free sandbox. Agents read the records in $96.9\%$ of open-ended runs and produce an artifact or a self-chosen task in $18.7\%$; with an explicit task, $86.0\%$ produce. Domain context does not change how many runs produce ($18.7\%$ vs.\ $16.0\%$), and the production ranking of the models largely carries over (Spearman $0.79$), while the ranking by whether a model reads the records does not ($0.19$). Context cuts boundary-probing turns per run by $60\%$, and some artifacts contradict the records. We contribute a test that a public-service institution can run on its own records before procurement; its rule-based taxonomy needs no labeled data, and scenarios, code, and trajectories are released.
BibTeX
@misc{kozachek2026read,
  author = {Diana Kozachek and Kinjalk Srivastava},
  title = {{Read and Ignore: What Self-Directed LLM Agents Do in Public Service Scenarios}},
  year = {2026},
  eprint = {2611.15460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15460},
}
Figure