TL;DR
EverydayBench is a new benchmark designed to evaluate how well AI agents complete complex, open-ended digital tasks on the live web. By moving away from fixed "correct" answers toward human-curated success criteria and real-time evidence verification, it reveals that even top-tier agents struggle to deliver truly complete, well-supported, and usable work.
What Problem Does This Paper Address?
Evaluating AI agents on the live web is notoriously difficult. Unlike static datasets, the web is dynamic: prices change, products go out of stock, and a single user request (like "plan a trip to Tokyo") can have dozens of valid outcomes.
Existing benchmarks often fall into two traps:
- Fixed-Answer Matching: They expect a specific string or value, which fails when the web updates or when multiple valid solutions exist.
- Holistic Judging: They use an LLM to give a "vibes-based" score, which often misses subtle failures like unsupported factual claims or violated constraints.
The authors argue that we need to evaluate satisfactory task completion—an outcome-level assessment of whether the agent’s work actually fulfills the user's request, including implicit needs that aren't always spelled out.
Methods
The core of EverydayBench is its Success-Criteria Curation (Figure 2). For each task, human curators define a set of explicit and implicit requirements. Evaluation then follows a three-pronged approach:
- Deterministic Direct Tests: Automated checks for hard constraints like "Price < $75" or verifying that a required JSON file has the correct schema.
- Source-Grounded Checks: The evaluator retrieves the specific webpages the agent cited at the time of the run and verifies if the claims made in the deliverable are actually supported by those pages.
- Rubric-based Component Scoring: An LLM judge (GPT-5.4) uses task-specific, human-authored rubrics to score qualitative aspects (e.g., "Does this travel itinerary account for the colder climate of the destination?").
This framework allows for "Open Outcomes"—the agent is free to choose any valid path or product, as long as it meets the predefined criteria.
Datasets
The paper introduces EverydayBench, a benchmark consisting of 77 human-authored tasks spanning eight domains:
- Consumer products & shopping
- Travel & local services
- Food, culture & entertainment
- Research
- Education & careers
- Business
- Health & public affairs
- Software & data work
The authors provide a detailed comparison (Table 1) showing how EverydayBench differs from existing benchmarks like GAIA or WebArena by combining live-web interaction, multi-format deliverables, and human-curated per-task rubrics.
Key Insights and Ideas
- The "Intent Recognition" Gap: A major reason agents fail is that they don't "look into the need." For example, if asked to find a short-term rental, an agent might find a great house but fail to check if local zoning laws actually allow short-term rentals in that area.
- Research vs. Delivery Failures: High-performing "frontier" models most often fail by overclaiming—stating details as facts that aren't actually on the cited source page. Lower-cost models, conversely, struggle with delivery—failing to follow file formats or missing requested items entirely.
- Temporal Sensitivity is Real: The authors found that scores for fixed agent outputs naturally decline over time (about 8% over 30 days) because the web content they rely on changes. This highlights why "live" evaluation is essential but also why it's a moving target.
- Harness Matters: The agentic "harness" (the code that manages tools and planning) is just as important as the underlying model. Using the same model in different harnesses led to pass rate swings of over 20%.
Key Results
The researchers evaluated 23 different agent-model configurations.
- Low Success Rates: Across all 1,771 runs, only 32.30% met the benchmark's strict completion standard.
- Top Performers: Claude Opus 5.5 and Claude Fable 5.1 tied for the highest strict pass rate at 54.55%.
- Reasoning Effort: Increasing "reasoning effort" (via a GPT-6 Sol sweep) improved performance from ~30% to ~45%, but the gains were non-monotonic, suggesting that just throwing more compute at a task doesn't guarantee a better outcome.
- Judge Reliability: The automated evaluation protocol showed high agreement with human experts ($\kappa = 0.695$ for grounding, $\kappa = 0.644$ for rubrics), outperforming simpler "one-prompt" LLM judges.
Limitations and Future Work
- Judge Dependency: The system relies on a powerful backbone model (GPT-5.4) for judging. While validated against humans, any bias in the judge could affect the benchmark.
- Scaling Curation: The per-task human curation is labor-intensive. Finding ways to automate the generation of high-quality success criteria while maintaining rigor is a key challenge.
- Reproducibility: Because the web is live, exact "bit-for-bit" reproducibility is impossible. The authors mitigate this by showing that relative rankings of agents remain stable over time (Figure 3).
- Future Directions: The authors suggest focusing on improving agentic "intent recognition"—teaching models to proactively identify and verify the implicit requirements of a user's goal.
Summary updated