EverydayBench: Evaluating Satisfactory Completion of Live User-Facing Digital Work

Fangrui Huang, Bingyang Wang, Chuhan Li, Yijiang Li, Arpandeep Khatua, Austin Shen, Riki R Z Choi, Cuiyuanxiu Chen, Jin Huang, Zachary Robertson, Kate Lin, Akshay Paruchuri, Sijun Tan, Xin Liu, Zhengmin Ni, Sanmi Koyejo, Ehsan AdeliarXiv 2026 · arXiv

TL;DR

EverydayBench is a new benchmark designed to evaluate how well AI agents complete complex, open-ended digital tasks on the live web. By moving away from fixed "correct" answers toward human-curated success criteria and real-time evidence verification, it reveals that even top-tier agents struggle to deliver truly complete, well-supported, and usable work.

What Problem Does This Paper Address?

Evaluating AI agents on the live web is notoriously difficult. Unlike static datasets, the web is dynamic: prices change, products go out of stock, and a single user request (like "plan a trip to Tokyo") can have dozens of valid outcomes.

Existing benchmarks often fall into two traps:

  1. Fixed-Answer Matching: They expect a specific string or value, which fails when the web updates or when multiple valid solutions exist.
  2. Holistic Judging: They use an LLM to give a "vibes-based" score, which often misses subtle failures like unsupported factual claims or violated constraints.

The authors argue that we need to evaluate satisfactory task completion—an outcome-level assessment of whether the agent’s work actually fulfills the user's request, including implicit needs that aren't always spelled out.

Figure 2: Standard fixed-answer curation and our success-criteria curation. Human curators translate an open-ended user request into criterion-level direct tests, grounding requirements, and weighted 1/3/5 rubric anchors, allowing multiple valid outcomes while making specific failures observable.

Methods

The core of EverydayBench is its Success-Criteria Curation (Figure 2). For each task, human curators define a set of explicit and implicit requirements. Evaluation then follows a three-pronged approach:

  1. Deterministic Direct Tests: Automated checks for hard constraints like "Price < $75" or verifying that a required JSON file has the correct schema.
  2. Source-Grounded Checks: The evaluator retrieves the specific webpages the agent cited at the time of the run and verifies if the claims made in the deliverable are actually supported by those pages.
  3. Rubric-based Component Scoring: An LLM judge (GPT-5.4) uses task-specific, human-authored rubrics to score qualitative aspects (e.g., "Does this travel itinerary account for the colder climate of the destination?").

This framework allows for "Open Outcomes"—the agent is free to choose any valid path or product, as long as it meets the predefined criteria.

Figure 1: Overview of the EverydayBench framework. A user request is paired with fixed success criteria; agents may submit different outcomes, which are evaluated through deterministic direct tests, source-grounded checks, and rubric-based component scoring.

Datasets

The paper introduces EverydayBench, a benchmark consisting of 77 human-authored tasks spanning eight domains:

  • Consumer products & shopping
  • Travel & local services
  • Food, culture & entertainment
  • Research
  • Education & careers
  • Business
  • Health & public affairs
  • Software & data work

The authors provide a detailed comparison (Table 1) showing how EverydayBench differs from existing benchmarks like GAIA or WebArena by combining live-web interaction, multi-format deliverables, and human-curated per-task rubrics.

Table 1: Comparison of representative AI agent benchmarks, grouped by evaluation paradigm. Everyday Tasks covers users’ everyday personal or workplace requests. Real Web indicates interaction with live real-world websites. Open Outcomes indicates that evaluation accepts substantively different valid results. Deliverable Form is the scored output: Answer (short or fixed-field answer), State (environment state), File (edited or generated file), Trajectory (action history and screenshots), Report (long-form text report), and Multi-format (a structured outcome together with user-facing reports, plans, tables, files, or tools). Executable Constraint Checks uses task-specific deterministic checks of output constraints, environment states, or artifact properties, beyond reference-answer or recorded-action matching alone. Source Verification assesses whether cited evidence supports submitted claims. Human-Curated Per- Task Rubrics indicates that human curators author judging rubrics or criteria individually for each task, rather than relying on answer matching, a generic judging prompt, or LLM-generated criteria. Evaluation columns refer to task-scoring protocols, excluding auxiliary analyses. ✓= yes, ✓

Key Insights and Ideas

  • The "Intent Recognition" Gap: A major reason agents fail is that they don't "look into the need." For example, if asked to find a short-term rental, an agent might find a great house but fail to check if local zoning laws actually allow short-term rentals in that area.
  • Research vs. Delivery Failures: High-performing "frontier" models most often fail by overclaiming—stating details as facts that aren't actually on the cited source page. Lower-cost models, conversely, struggle with delivery—failing to follow file formats or missing requested items entirely.
  • Temporal Sensitivity is Real: The authors found that scores for fixed agent outputs naturally decline over time (about 8% over 30 days) because the web content they rely on changes. This highlights why "live" evaluation is essential but also why it's a moving target.
  • Harness Matters: The agentic "harness" (the code that manages tools and planning) is just as important as the underlying model. Using the same model in different harnesses led to pass rate swings of over 20%.

Key Results

The researchers evaluated 23 different agent-model configurations.

  • Low Success Rates: Across all 1,771 runs, only 32.30% met the benchmark's strict completion standard.
  • Top Performers: Claude Opus 5.5 and Claude Fable 5.1 tied for the highest strict pass rate at 54.55%.
  • Reasoning Effort: Increasing "reasoning effort" (via a GPT-6 Sol sweep) improved performance from ~30% to ~45%, but the gains were non-monotonic, suggesting that just throwing more compute at a task doesn't guarantee a better outcome.
  • Judge Reliability: The automated evaluation protocol showed high agreement with human experts ($\kappa = 0.695$ for grounding, $\kappa = 0.644$ for rubrics), outperforming simpler "one-prompt" LLM judges.
Stability of the EverydayBench judge compared to ablated baselines.

Limitations and Future Work

  • Judge Dependency: The system relies on a powerful backbone model (GPT-5.4) for judging. While validated against humans, any bias in the judge could affect the benchmark.
  • Scaling Curation: The per-task human curation is labor-intensive. Finding ways to automate the generation of high-quality success criteria while maintaining rigor is a key challenge.
  • Reproducibility: Because the web is live, exact "bit-for-bit" reproducibility is impossible. The authors mitigate this by showing that relative rankings of agents remain stable over time (Figure 3).
  • Future Directions: The authors suggest focusing on improving agentic "intent recognition"—teaching models to proactively identify and verify the implicit requirements of a user's goal.
Figure 3: Panel (a) compares multiple agent runs with near-synchronized judging at different points in time to validate benchmark stability; panel (b) holds agent run fixed to show time-sensitivity to web contents under near-synchronized, 20-day- delayed, and 30-day-delayed judging.

Summary updated

Abstract

Everyday digital tasks, from selecting products and planning trips to conducting research and creating artifacts, require AI agents to deliver useful outcomes rather than merely retrieve web information. Yet evaluating these outcomes on the live web is challenging, as a single request may admit multiple valid results, and changing web information can alter which results remain valid over time. Although existing benchmarks address aspects of this setting, they do not fully integrate predefined success criteria, current evidence verification, and deliverable-quality assessment for agent-selected outcomes. To address this gap, we introduce EverydayBench, a benchmark of 77 human-authored tasks spanning eight domains of everyday life and work. Each task specifies success criteria before execution without prescribing an answer or workflow, allowing agents to select their own outcomes. To evaluate submissions under fixed success criteria, EverydayBench combines (1) deterministic checks of explicit constraints, (2) verification of claims against agent-cited webpages retrieved at evaluation time, and (3) rubric-based assessment of final deliverable quality. We further validate the protocol through human judgments, repeated scoring under fixed evidence, and controlled grounding failures. Across 23 reported agent-model configurations, Claude Opus 5.5 and Claude Fable 5.1 tie for the highest strict pass rate at 54.55%; a controlled GPT-6 Sol effort sweep ranges from 29.87% at low effort to 45.45% at max effort. Reliability varies substantially by task type: GPT-5.6 Sol leads lookup, Fable 5.1 leads planning, Opus 5.5 and Fable 5.1 tie on comparison, and GPT-6 Astra and Sonnet 5.5 tie on customized deliverables. Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests.
BibTeX
@misc{huang2026everydaybench,
  author = {Fangrui Huang and Bingyang Wang and Chuhan Li and Yijiang Li and Arpandeep Khatua and Austin Shen and Riki R Z Choi and Cuiyuanxiu Chen and Jin Huang and Zachary Robertson and Kate Lin and Akshay Paruchuri and Sijun Tan and Xin Liu and Zhengmin Ni and Sanmi Koyejo and Ehsan Adeli},
  title = {{EverydayBench: Evaluating Satisfactory Completion of Live User-Facing Digital Work}},
  year = {2026},
  eprint = {2611.25146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25146},
}
Figure