From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production

Segev Shlomov, Alon Oved, Sami Marreed, Ido Levy, Offer Akrabi, Avi Yaeli, Lukasz Strak, Elizabeth Koumpan, Yinon Goldshtein, Eilam Shapira, Nir Mashkif, Asaf Adi

Abstract

Agents are rapidly advancing in automating digital work, but enterprises face a harder challenge: moving beyond prototypes to deployed systems that deliver measurable business value. This path is complicated by fragmented frameworks, slow development, and the absence of standardized evaluation practices. Generalist agents have emerged as a promising direction, excelling on academic benchmarks and offering flexibility across task types, applications, and modalities. Yet, evidence of their use in production enterprise settings remains limited. This paper reports IBM's experience developing and piloting the Computer Using Generalist Agent (CUGA), which has been open-sourced for the community. CUGA adopts a hierarchical planner–executor architecture with strong analytical foundations, achieving state-of-the-art performance on AppWorld and WebArena. Beyond benchmarks, it was evaluated in a pilot within the Business-Process-Outsourcing talent acquisition domain, addressing enterprise requirements for scalability, auditability, safety, and governance. To support assessment, we introduce BPO-TA, a 26-task benchmark spanning 13 analytics endpoints. In preliminary evaluations, CUGA approached the accuracy of specialized agents while indicating potential for reducing development time and cost. Our contribution is twofold: presenting early evidence of generalist agents operating at enterprise scale, and distilling technical and organizational lessons from this initial pilot.

Introduction: The Enterprise Production Gap

While AI agents are rapidly advancing in research settings, most remain prototypes that excel on synthetic benchmarks but struggle with the rigorous demands of enterprise production. Organizations face a "production gap" characterized by fragmented frameworks, slow development cycles, and a lack of auditability. The central challenge is productization: transforming a generalist agent—designed to handle diverse tasks across web portals, APIs, and spreadsheets—into a system reliable enough for a high-stakes business environment. Enterprises require strict adherence to Service Level Agreements (SLAs), robust governance to prevent hallucinations, and clear provenance to ensure data integrity.

The CUGA Architecture: Hierarchical Planning and Execution

To bridge this gap, IBM researchers developed CUGA (Computer Using Generalist Agent). CUGA utilizes a hierarchical planner–executor architecture organized into three distinct control layers designed for traceability and reliability:

  1. Chat/Context Layer: This layer preprocesses user intent and manages message history to provide a coherent context for the agent.
  2. Outer Loop (Task Planning): A central component of CUGA is the persistent task ledger. This ledger decomposes complex requests into manageable sub-tasks, providing an audit trail and allowing the agent to recover from partial failures by "remembering" its state across multi-step reasoning.
  3. Inner Loop (Execution): Specialized sub-agents—including API, Web, CLI, and File-system agents—execute specific sub-tasks as directed by the planner.
The CUGA hierarchical architecture, featuring context enrichment, a plan controller agent, and specialized sub-agents.

To ensure enterprise-grade reliability, CUGA incorporates several "safety-first" mechanisms. Schema-Grounded Prompting enforces strict JSON I/O to prevent parsing errors, while Reflective Retries allow the agent to self-correct if a tool response deviates from the expected schema. Furthermore, lightweight data manipulations are performed in a Sandboxed Computation environment (the "Code Agent") to isolate file and network access, and Provenance Logging ensures every response includes a detailed record of API paths and computation logs for auditing.

Evaluation and Enterprise Benchmarking

CUGA was evaluated across three major benchmarks to test its versatility and enterprise readiness:

  • WebArena: A realistic web environment for browser-based agents.
  • AppWorld: A benchmark for multi-application orchestration via programmatic APIs.
  • BPO-TA (New): The authors introduced the Business Process-Outsourcing Talent Acquisition benchmark. It consists of 26 decision-support tasks across 13 read-only analytics endpoints, specifically designed to test enterprise-specific needs like cross-API joins and the graceful handling of unsupported queries.

Key Results and Business Impact

CUGA achieved state-of-the-art performance, reaching a 61.7% success rate on WebArena and 48.2% on AppWorld, ranking first among published agents at the time of writing. On the enterprise-specific BPO-TA benchmark, CUGA reached an impressive 87% accuracy.

Performance metrics across different task levels, showing success rates for task and scenario goals.

Beyond accuracy, the pilot demonstrated significant business ROI. The use of a generalist agent led to a 90% reduction in development time and a 50% reduction in development cost compared to building task-specific specialized agents. For end-users, such as recruiters, the agent reduced the time needed for complex manual data pulls from approximately 20 minutes to just 2–5 minutes.

Lessons from the Field: Governance and Scalability

The pilot yielded several critical insights for enterprise AI deployment:

  • Generalization Reduces Technical Debt: A single generalist architecture can approach the accuracy of hand-crafted agents while being significantly easier to maintain and scale.
  • The Power of the Ledger: The persistent plan ledger was essential, reducing failures by over one-third by maintaining state across complex reasoning chains.
  • Governance as a Feature: In the BPO pilot, the agent was configured to decline unsupported requests rather than hallucinate. This "graceful failure" significantly increased user trust.
  • Onboarding Speed: By using a centralized API/Tool Hub to canonicalize OpenAPI specs into LLM-friendly schemas, the team reduced the time to onboard new analytics endpoints from weeks to hours.

Conclusion and Future Work

While the current pilot was restricted to read-only APIs to minimize risk, it proves the viability of generalist agents in production. Future work will focus on moving toward "create/update" capabilities, which will require even more stringent safety guardrails. Additionally, the team plans to explore configurable Human-in-the-Loop (HITL) mechanisms, model optimization for routine sub-tasks, and more explicit natural language policy enforcement.

Summary updated

Abstract

Agents are rapidly advancing in automating digital work, but enterprises face a harder challenge: moving beyond prototypes to deployed systems that deliver measurable business value. This path is complicated by fragmented frameworks, slow development, and the absence of standardized evaluation practices. Generalist agents have emerged as a promising direction, excelling on academic benchmarks and offering flexibility across task types, applications, and modalities. Yet, evidence of their use in production enterprise settings remains limited. This paper reports IBM's experience developing and piloting the Computer Using Generalist Agent (CUGA), which has been open-sourced for the community (https://github.com/cuga-project/cuga-agent). CUGA adopts a hierarchical planner--executor architecture with strong analytical foundations, achieving state-of-the-art performance on AppWorld and WebArena. Beyond benchmarks, it was evaluated in a pilot within the Business-Process-Outsourcing talent acquisition domain, addressing enterprise requirements for scalability, auditability, safety, and governance. To support assessment, we introduce BPO-TA, a 26-task benchmark spanning 13 analytics endpoints. In preliminary evaluations, CUGA approached the accuracy of specialized agents while indicating potential for reducing development time and cost. Our contribution is twofold: presenting early evidence of generalist agents operating at enterprise scale, and distilling technical and organizational lessons from this initial pilot. We outline requirements and next steps for advancing research-grade architectures like CUGA into robust, enterprise-ready systems.
BibTeX
@inproceedings{shlomov2026from,
  author = {Segev Shlomov and Alon Oved and Sami Marreed and Ido Levy and Offer Akrabi and Avi Yaeli and Lukasz Strak and Elizabeth Koumpan and Yinon Goldshtein and Eilam Shapira and Nir Mashkif and Asaf Adi},
  title = {{From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i47.41485},
}
Figure