TL;DR
While AI models are crushing technical benchmarks in coding, law, and math, they remain largely untested in the "organizational tissue" that defines real-world work. This paper introduces a taxonomy for organizational task difficulty—the complexity arising from interdependence, authority, and conflicting information—and reveals that fewer than one-quarter of prominent AI benchmarks currently measure these critical capabilities.
What Problem Does This Paper Address?
We are currently witnessing a paradox: AI capabilities are skyrocketing on paper, yet their impact on actual economic productivity remains uneven. The authors argue this is because our benchmarks focus almost exclusively on technical task difficulty (solving a well-defined problem in isolation) while ignoring organizational task difficulty.
In a real company, a task is rarely "atomic." Success depends on navigating a web of relations: waiting for a manager’s approval, reconciling conflicting data from different departments, or adjusting a plan when a colleague misses a deadline. Current benchmarks treat AI like a solo genius in a dark room, but economically valuable work happens in "embedded" environments. Without measuring an agent's organizational navigation capability, we cannot accurately predict how AI will perform—or fail—within a firm.
Methods
The authors bridge the gap between AI evaluation and classical organization theory. They propose a four-layer taxonomy to describe how work is embedded in an organization:
To test the state of the field, they used a Large Language Model (Claude Opus 5.5) to audit 522 prominent benchmarks collected from industry indices, top-tier research conferences (like NeurIPS and ICML), and lab model cards. They coded each benchmark based on whether it mentioned, claimed to test, or actually measured six specific dimensions of organizational structure:
- Task interdependence: How much the task depends on other activities.
- Information structure: How knowledge is distributed or if it's contradictory.
- Authority structure: Who has the right to decide or approve.
- Temporal structure: Deadlines, sequencing, and latency.
- Incentive structure: Alignment or conflict between different actors' goals.
- Environmental dynamism: How much the "rules of the game" change mid-task.
Datasets
The study analyzed a massive meta-dataset of existing benchmarks rather than training a new model. The sample included:
- Industry Indices: Benchmarks from Artificial Analysis and Epoch AI.
- Top-Conference Papers: Award-winning or spotlight papers from NeurIPS, ICML, ICLR, ACL, EMNLP, and NAACL (2023–2026).
- Lab Model Cards: Benchmarks reported by major labs like OpenAI, Anthropic, and Google.
They highlighted a few "organizational-aware" benchmarks as examples, such as $\tau^2$-bench (customer service with simulated users) and TheAgentCompany (AI working as an employee in a simulated firm).
Key Insights and Ideas
- Technical vs. Organizational Difficulty are Independent: A task can be technically trivial but organizationally a nightmare. The authors illustrate this with a 2x2 matrix, noting that a nurse administering a routine dose of medication is technically simple but organizationally difficult due to the required chain of physician orders and pharmacist checks.
- The "Null Value" Assumption: Most current benchmarks hold organizational conditions at a "null value." By providing all necessary info upfront and granting the agent total authority, they remove the very conditions that make real work hard.
- Organizational Navigation Capability: The authors define this as a distinct AI skill—the ability to recognize how actions are embedded in a changing structure and adapt accordingly. This is separate from "conduct" (following rules).
- Concentration of Complexity: When benchmarks do include organizational elements, they are highly concentrated. "Information structure" (reconciling documents) is common, while "Authority structure" and "Task interdependence" are nearly non-existent in prominent benchmarks.
Key Results
- The Gap is Wide: Only 23% (118 of 522) of prominent benchmarks claim to represent any of the six organizational dimensions.
- Measurement is Rare: Even fewer—only 16%—actually provide a separate metric to measure how well the AI handles these organizational challenges.
- A Growing Trend: There is a clear upward trajectory. Only 2% of benchmarks published before 2023 claimed an organizational dimension, compared to 33% of those published in the first ten months of 2026. This reflects the industry's shift toward "agentic" AI.
- Simulated Colleagues are Missing: In the rare cases where task interdependence was claimed, only two prominent benchmarks actually involved "colleagues" in an organization; most others relied on simple human-collaborator or game-based setups.
Limitations and Future Work
The authors acknowledge that their taxonomy is rooted in human-centered organization theory. It remains an open question whether "agentic organizations" (swarms of AI agents working together) will follow the same rules of bounded rationality and hierarchy that humans do.
Furthermore, the study relied on what was reported in research papers. Some benchmarks might contain organizational complexity that isn't explicitly described in the text, leading to a potential undercount.
Future Work: The authors call for a new generation of benchmarks that hold technical difficulty constant while varying organizational dimensions. This would allow researchers to "stress test" an AI's ability to handle a difficult boss or a messy information environment without conflating it with the AI's ability to write code or solve math.
Summary updated