TL;DR
The researchers introduce SWE-CoWork, a new benchmark that evaluates AI coding agents as "digital employees" within a simulated software organization. While modern models like GPT-6 Astra and Claude-Opus-5.5 are becoming proficient at isolated coding tasks, they struggle significantly with the organizational overhead of real-world engineering—collaborating with colleagues, resolving conflicting documentation, and tracking requirements through to delivery.
What Problem Does This Paper Address?
Current AI coding benchmarks (like the original SWE-bench) typically present agents with a "pre-scoped" task: a static issue description and a repository. In reality, software engineering is a collaborative, messy endeavor. Requirements are rarely found in a single place; they are scattered across chat logs, project boards, and the heads of various colleagues.
The authors argue that current agents act as passive "spec executors." They can write code if told exactly what to do, but they lack the "organizational intelligence" to:
- Recover requirements from diverse stakeholders.
- Reconcile conflicting evidence (e.g., an outdated design doc vs. a recent chat message).
- Proactively follow up on deferred questions.
- Verify and hand off a complete solution rather than just a partial patch.
Methods
The authors developed SWE-CoWork, a benchmark that places an agent into a simulated engineering organization.
- The Environment: Each task includes a real open-source repository and a workplace populated by 26 digital employees. These are LLM-backed (GPT-5.6 Terra) personas with distinct roles (e.g., module maintainers, release coordinators), bounded knowledge, and specific collaboration behaviors.
- The Workflow: Agents start with a brief "handoff" note. They must use a tool interface to search directories, inspect project boards, read chat histories, and message colleagues to uncover what actually needs to be built.
- Oracle-Spec Comparison: To isolate whether failures are due to finding information or using it, the authors created a paired "Oracle-Spec" condition where the agent is given all requirements upfront, bypassing the organizational search.
Datasets
- SWE-CoWork: 101 development assignments across 101 real repositories.
- Diversity: Covers 11 programming languages and 12 application domains (Web, Dev Tools, Databases, etc.).
- Scale: The benchmark includes over 40,000 chat messages, 10,000 board cards, and 5,000 versioned documents. A median task involves adding nearly 3,000 lines of code across 60 files.
Key Insights and Ideas
- The "Implementation Gap": Even when models are given the requirements perfectly (Oracle-Spec), their success rate for completing an entire assignment doesn't budge. This suggests that the bottleneck isn't just "finding the info"—it's the cognitive load of managing a large-scale implementation.
- Finding the Person is Easy, Getting the Info is Hard: Claude-Opus-5.5 correctly identifies the "accountable owner" for a requirement 93% of the time. However, it only manages to actually extract the necessary requirement from them 64% of the time.
- The Deferral Graveyard: When a simulated colleague says, "I'll get back to you," agents often fail to follow up. 153 out of 450 deferred threads were simply abandoned, even when the agent had hours of execution budget left.
- Hallucinated Completion: Perhaps most concerningly, 31% of requirements that agents reported as "done" in their final handoff actually failed their tests. Agents are overconfident in their own verification processes.
Key Results
The evaluation of eleven frontier models revealed a massive gap between "fixing a bug" and "delivering a task":
- Partial Success (F2P): Leading models like Claude-Opus-5.5 and GPT-6 Astra satisfy about 63% of individual "fail-to-pass" test nodes.
- Full Task Success (Pass Rate): These same models complete fewer than 8% of full assignments.
- Weaker Models: Models like DeepSeek V4 Pro or Seed 2.1 Pro struggled significantly more, often failing to send a single message to a colleague and obtaining less than 2% of the required information.
- Efficiency: There was little correlation between the number of "tool calls" (actions taken) and success. Some models exhausted their 8-hour budget with thousands of actions but achieved lower scores than more "efficient" models.
Limitations and Future Work
The authors acknowledge that while the simulated colleagues are rule-governed and LLM-backed, they may not perfectly capture the nuances of human frustration or ambiguity.
Future Work points toward:
- Explicit Requirement Tracking: Agents need internal "state machines" to track every requirement from discovery to verified code.
- Autonomous Teams: Extending the benchmark to evaluate teams of agents working together.
- Improved Verification: Closing the gap between an agent thinking it's done and the code actually passing private executable tests.
Summary updated