AutoBuddy: Evaluating Proactive, Always-On Agents on Task Discovery and Execution in a Simulated Workplace

Shiqi Chen, Daniel Madrigal Diaz, Wei-Ning Chen, Gbola Afonja, Fangkai Yang, Rujia Wang, Chetan Bansal, Victor Rühle, Saravan Rajmohan, Dongge HanarXiv 2026 · arXiv

This paper from Microsoft Research introduces AutoBuddy, a framework designed to move beyond "reactive" AI agents—which wait for a specific prompt—toward "proactive," always-on assistants that can independently discover and execute tasks within a complex workplace environment.

TL;DR

AutoBuddy-Bench is a novel simulator that generates a consistent, grounded workplace environment to evaluate how well LLM agents can proactively identify "hidden" tasks in emails and chats and then execute them. The study reveals that task discovery and execution are distinct capabilities, with even the most advanced models often failing to bridge the gap between finding a task and successfully completing it in business systems.

What Problem Does This Paper Address?

Most current AI agent benchmarks are instruction-driven: you tell the agent exactly what to do (e.g., "Book a meeting with Alice"). However, in a real office, work is rarely that tidy. Tasks are buried in a mountain of "noise"—unanswered emails, Slack threads, and calendar invites. A true digital assistant needs to be proactive, scanning the user's workspace to infer what needs to be done without being asked.

The authors identify two major hurdles:

  1. For the Agent: Distinguishing actual obligations from background noise (like a request already handled by a colleague).
  2. For Evaluation: Creating a benchmark that is "mutually consistent." If an agent sends an email, the simulated world needs to reflect that change accurately across all systems to verify the outcome.

Methods

The researchers developed two core components:

  • AutoBuddy-Bench: A simulator that builds a company from a short specification. It uses a temporal graph to track every employee, project, and event. When a business event occurs (e.g., a software training run finishes), the system generates a "ground-truth" obligation. This is then rendered into realistic artifacts like emails or chat messages that the agent must find.
  • The AutoBuddy Agent: A reference agent that operates in two distinct phases. First, it performs Task Discovery, mining the workspace to create a structured list of owed tasks. Second, it performs Task Execution, where it uses tools (CRM, Finance systems, etc.) in a fresh session to complete each identified task.
Figure 2: One task followed through AutoBuddy-Bench (Section 3.1). (1) A company scope fixes the business, headcount, period and systems; (2) positions are generated with dated roles and access rights; (3) events are generated and code computes the ground-truth outcome from the rule that governs them: three training runs finish, and the lab imposes a rule that runs may be compared only if they were trained on the same data. Run B has the lowest loss but was trained on more data, so the correct decision is to promote Run A, the best comparable run; (4) the engineer’s workspace is rendered as of the actor-day, with the ground truth and evaluation labels withheld. Steps 2 and 3 write to a temporal graph that can be queried as of any date. The illustrated company is an example, not the evaluation world.

Datasets

The primary dataset is the newly introduced AutoBuddy-Bench. For their evaluation, the authors generated an "evaluation world" simulating six months of company history, resulting in 120 ground-truth tasks across 94 actor-days and 44 different employees. These tasks span domains like HR, Finance, Sales, and Engineering research.

The benchmark was used to test 17 different LLM backbones, including models from Anthropic (Claude), OpenAI (GPT), DeepSeek, and others.

Key Insights and Ideas

  • Discovery and Execution are Decoupled: A model’s ability to find a task in a messy inbox (precision/recall) is only weakly correlated with its ability to actually do the work. This suggests that "proactivity" requires a different set of cognitive skills than "instruction following."
  • The "Stronger Model" Shift: There is a systematic shift in failure modes as models get more powerful. Weak models fail at discovery (they simply don't see the tasks). Mid-tier models find the tasks but stop short of execution. The strongest models find almost everything but fail during the complex logic of business-system operations.
  • The "Stop and Ask" Bottleneck: Interestingly, the most advanced models (like the GPT-6 series in this study) were often too cautious. They frequently identified tasks correctly but then "stopped to ask" for clarification instead of taking action, leading to lower end-to-end scores despite high discovery recall.
Figure 4: Where each model fails, by stage, in the order of Table 1. (a) Discovery errors per run over all 51 runs: ground-truth tasks never listed (left), and listed tasks with no ground-truth counterpart (FP, right), by what the record they cite is: work the workspace shows was already closed, a request addressed to a colleague, or background with no request. (b) Outcome of every executed (discovered) task over all runs; the number on each bar is the share receiving full credit, and each execution is assigned to the first category that applies, in the order shown.

Key Results

  • Leaderboard: Claude Opus 5.5 emerged as the top performer with an end-to-end score of 0.698, followed closely by Claude Opus 5 (0.680) and GPT-5.6-sol (0.654).
  • Performance Range: There is a massive gap between tiers; the weakest model (Claude Haiku 4.5) scored only 0.234.
  • Discovery Success: GPT-6.1-sol achieved the highest discovery recall (0.989), finding nearly every task, yet it only ranked 7th overall because it declined to finish many of them, often stopping to ask for more information.
Figure 1: End-to-end score of AutoBuddy with 17 backbone models on the same 120 tasks: mean over three runs.

Limitations and Future Work

The authors acknowledge that while their simulator is grounded in a temporal graph, it is still a simulation. Real-world workplace data is often even noisier and more ambiguous.

Future directions include:

  • Multi-Agent Interaction: Evaluating how proactive agents interact with each other or with a human-in-the-loop.
  • Long-Horizon Memory: Improving how agents track tasks that span weeks or months.
  • Skill Distillation: Further exploring the agent's ability to "distill" successful executions into reusable skills for future tasks.

Summary updated

Abstract

As LLM agents become more autonomous, they are increasingly expected to run as always-on assistants on a user's behalf, identifying work to be done and completing it without being asked. Yet existing benchmarks usually give the agent its task, whereas in knowledge work the requests sit in the user's workspace and the agent must first recognise which obligations exist. We study proactive task discovery and execution, in which an agent must infer a user's pending tasks from their workspace and complete them in the systems where the work lives. This setting poses two challenges. For the agent, the tasks are buried in a large volume of mail, chat, documents and records. For evaluation, a benchmark needs a simulated organisation whose people, events and documents are mutually consistent, so that tasks arise realistically and their correct outcomes are known. We introduce AutoBuddy-Bench, a simulator that addresses these challenges. It models an organisation's employees, projects, business systems and event history as a temporal graph, renders it into realistic workspace artefacts, derives ground-truth tasks from the events that create obligations, and computes each task's outcome from the world state; execution is judged by the agent's changes to the workspace. To evaluate current LLMs, we build AutoBuddy, a reference agent with interchangeable backbone LLMs: it identifies tasks from the workspace, grounding each in its source records, and executes each in a separate session with write access. Varying only the backbone, we find a systematic shift in where tasks are lost: weaker models fail mainly at discovery, intermediate models identify tasks but often stop short of carrying them out, and the strongest models fail mainly during execution, most often on business-system operations. Discovery precision is only weakly related to execution quality, suggesting the two are distinct capabilities.
BibTeX
@misc{chen2026autobuddy,
  author = {Shiqi Chen and Daniel Madrigal Diaz and Wei-Ning Chen and Gbola Afonja and Fangkai Yang and Rujia Wang and Chetan Bansal and Victor Rühle and Saravan Rajmohan and Dongge Han},
  title = {{AutoBuddy: Evaluating Proactive, Always-On Agents on Task Discovery and Execution in a Simulated Workplace}},
  year = {2026},
  eprint = {2611.12393},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12393},
}
Figure