TL;DR
This paper introduces a new diagnostic benchmark to evaluate how LLM agents behave when given "underspecified" instructions (e.g., "see where we stand") in realistic public-service scenarios. The researchers found that while agents are diligent readers, they rarely take initiative to act unless given an explicit task—and when they do act, they often skip the reading or produce reports that contradict the source records.
What Problem Does This Paper Address?
In public service sectors like healthcare and governance, staff often give vague, open-ended instructions to assistants. As we move toward deploying LLM agents in these roles, a critical question arises: what does an agent do with its "discretion"?
Existing benchmarks generally fall into two camps:
- Task-oriented: Measuring if an agent can complete a highly specific goal (like fixing a GitHub issue).
- Empty Sandboxes: Observing agent behavior in "context-free" environments where there is nothing to actually do.
Neither captures the reality of a workplace where an agent has access to a rich file tree of records, stakeholders, and procedures but no clear "to-do" list. This paper addresses the lack of evaluation for self-directed behavior—the ability of an agent to identify what needs doing and execute it accurately without being micro-managed.
Methods
The authors developed four expert-validated scenarios representing recurring operational cycles:
- Governance: Participatory budgeting.
- Healthcare: Ward planning.
- Education: Course catalog assembly.
- Environmentalism: Waterway monitoring.
Each scenario is a static file tree containing "decision levers"—intentional gaps, inconsistencies, or equity shortfalls that an intelligent agent should notice.
The study collected 2,400 trajectories from ten different LLMs (ranging from open-weight models like Llama-3.1-8B to frontier APIs like GPT-4o-mini) under four prompt conditions, ranging from total "freedom" to an explicit "task baseline." To classify behavior, they used a rule-based taxonomy of 9 labels and 22 subcategories, allowing for automated, reproducible analysis of nearly 94,000 individual turns.
Datasets
- Public Service Scenarios: Four new, expert-validated directory trees with synthetic records, procedures, and timelines.
- Trajectory Dataset: 2,400 runs (scenario-based) plus 600 directed runs.
- Context-Free Baseline: 1,000 runs in an empty sandbox from prior work used for comparative analysis.
Key Insights and Ideas
- The Discretion Paradox: When given freedom, agents read almost everything (96.9% of runs) but act on almost nothing (only 18.7% produce an artifact). Conversely, when given an explicit task, they act (86%) but often stop reading the context (only 50% read the domain files).
- Context as a Stabilizer: Adding domain context (files and records) reduces "boundary probing"—the tendency of agents to test the limits of their environment or try to "escape" the sandbox—by 60%.
- Model "Personalities": Models differ more from each other than they do across different domains. As seen in Figure 1, runs of the same model tend to cluster together in the output space, showing consistent "behavioral signatures" like aimless wandering, task looping, or deliberate auditing.
- The Groundedness Gap: Even when agents produce reports, they are not always reliable. 1.4% of open-ended runs produced artifacts that directly contradicted the records (e.g., citing wrong budget figures). For some frontier models, this "contradiction rate" was as high as 35% in directed tasks.
Key Results
The researchers found that a model's tendency to produce work is surprisingly consistent across environments. The ranking of which models were most "productive" in a context-free sandbox correlated strongly (Spearman 0.79) with their productivity in complex scenarios.
However, the reading behavior did not carry over (correlation of only 0.19). A model that reads a lot in an empty sandbox might ignore the records entirely when placed in a real-world scenario.
Quantitatively, "Environment Exploration" dominated the agents' time, accounting for 84.1% of all turns, while actual "Artifact Creation" was a mere 3.7%.
Limitations and Future Work
- Static Environments: The scenarios are "frozen" in time. In a real office, files change, colleagues message you, and deadlines shift. The authors acknowledge that agents might look more active in a dynamic environment.
- No Social Interaction: The current action set does not allow agents to ask a human for clarification, which is a standard behavior for a real-world assistant.
- Rule-Based Coarseness: While the rule-based taxonomy is highly reproducible, it is less nuanced than human expert judgment, particularly regarding the quality or substance of the work produced.
- Future Directions: The authors suggest this benchmark should be used as a pre-procurement "audit" for public institutions to test how specific models handle their own internal records before deployment.
Summary updated