A Capability Framework for Evaluating Agentic AI Platforms in Regulated Enterprises

Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona ZahidarXiv 2026 · arXiv

TL;DR

This paper introduces a rigorous capability framework for evaluating "Agentic AI" platforms—systems where LLMs don't just chat, but plan and execute state-changing actions—within the strict constraints of regulated industries like banking. By analyzing an industrial "bakeoff" of ten major platforms, the authors expose a significant "halo effect" in current evaluation practices and propose a new, four-layered "gate-then-score" protocol to move from subjective judgment to objective measurement.

What Problem Does This Paper Address?

Enterprises are currently procuring agentic AI platforms faster than they can effectively measure them. While public benchmarks like AgentBench or GAIA exist, they primarily score the underlying LLM in a fixed environment. However, an enterprise doesn't just buy a model; it buys an integrated platform where the orchestration, memory management, and governance layers often dictate the final outcome more than the model itself.

Furthermore, regulated institutions (like the global bank featured in this study) face unique hurdles: they must satisfy strict auditability, accountability, and model-risk obligations. Currently, buyers often fall back on vendor demonstrations, which risks mistaking "presentation quality" for "engineering quality." The authors aim to bridge this gap by defining what makes an agentic platform "enterprise-ready."

Figure 1. The automation continuum. Rightward movement in- creases run-time autonomy, the scope of delegated decisions and, consequently, the governance burden.

Methods

The researchers conducted an "Agentic AI Bakeoff" involving ten platforms, ranging from open-source frameworks to hyperscaler ecosystems (e.g., Google Vertex AI, Microsoft AutoGen, H2O.ai).

The evaluation was structured around:

  1. A Nine-Dimension Taxonomy: Platforms were scored on Reflection, Tool Use, Memory, Planning, Multi-agent coordination, Ease of Use, Protocol Support, Quantitative Performance, and Evaluation/Explainability.
  2. Three Financial Workloads: Counterparty credit risk (CCR), Retrieval-Augmented Generation (RAG), and Process Automation.
  3. A Multi-Stage Protocol: Moving from organizational screening to deep-dive sessions and live "bakeoff" demos.
  4. Psychometric Analysis: The authors used Cronbach’s $\alpha$ to test the internal consistency of the scores, revealing whether raters were truly distinguishing between technical dimensions or just following a general impression.

Datasets

Rather than using standard public datasets, the study employed three bespoke financial-services scenarios designed with subject matter experts (SMEs):

  • Counterparty Credit Risk (CCR): A complex task combining structured exposure data, unstructured regulatory documentation, and logic.
  • RAG: A standard enterprise workload focused on retrieval, grounding, and verifiability.
  • Process Automation: Testing goal attainment from minimal instruction, involving multi-step orchestration and exception handling.

Key Insights and Ideas

  • The "Halo Effect": The most striking finding was a Cronbach’s $\alpha$ of 0.98 across nominally distinct dimensions. This suggests a strong "halo effect"—raters who liked a platform's demo tended to give it high scores across all technical categories, regardless of specific performance. This highlights the danger of demonstration-based scoring.
  • Orchestration > Model: In an enterprise context, the "governance and control plane" (Figure 2) is the most critical component. A platform's ability to handle tool failures, maintain persistent memory, and follow safety guardrails is more important than the raw reasoning power of the LLM.
  • Workload-Specific Strengths: Aggregated scores often hide specialized capabilities. For example, the platform Letta excelled in memory management (D3), while Microsoft led in protocol support (D7), even if they weren't the overall winners.
  • Gate-then-Score: The authors propose that evaluation should be non-compensatory. A platform shouldn't be able to "average out" a failure in a critical security gate (like prompt-injection resistance) with a high score in "ease of use."
Figure 2. Reference architecture for an enterprise agentic platform. The dashed boundary is the governance and control plane that regulated deployments must wrap around every component.

Key Results

The bakeoff revealed a clear three-tier structure in the market:

  • Tier 1 (Top Performers): H2O.ai, Google, Microsoft, Writer, and Letta. These platforms showed robust capabilities across most dimensions.
  • The Gap: There was a massive 18.7-point gap between the bottom of Tier 1 and the top of Tier 2, suggesting a significant maturity divide in the current vendor landscape.
  • Stability: These tiers remained stable even when the authors applied different weightings (e.g., a "governance-first" vs. a "developer-first" profile).
Heat map of mean panel scores across the nine dimensions, showing the three-tier structure.

Limitations and Future Work

The authors acknowledge several limitations:

  • Assisted Conditions: Vendors helped prepare the demos, so the results reflect "demonstrated capability" rather than "out-of-the-box" performance.
  • Subjectivity: Despite the taxonomy, ratings remain subjective. To fix this, the authors propose a "Stage 5" (Figure 3) involving buyer-run harnesses and repeated, automated runs to generate trace-level reliability metrics.
  • Future Directions: The paper calls for a research agenda focused on "adversarial variants" (e.g., indirect injection probes) and the development of a "business-value identity" to measure the actual ROI of agentic automation beyond just time saved.
Figure 3. Multi-stage evaluation protocol. Each stage narrows the candidate pool while increasing evidential depth and per-vendor cost. Solid: executed; dashed: proposed extension.

Summary updated

Abstract

Agentic AI platforms-systems in which LLM agents plan, invoke tools, retain memory and coordinate to change the state of enterprise systems-are being procured faster than they can be measured. Public agent benchmarks score models in fixed harnesses; enterprises buy integrated platforms whose orchestration, memory, governance and tooling layers dominate outcomes, and they buy them largely on the strength of vendor demonstrations. We report an industrial evaluation programme ("bakeoff") run by the innovation laboratory of a global bank in which ten platforms-spanning open-source frameworks, proprietary products and hyperscaler ecosystems-were assessed by a cross-functional panel on nine capability dimensions across three financial-services workloads: counterparty credit risk, retrieval-augmented generation and process automation. Scores separate the field into three tiers that are stable under alternative weightings, and surface workload-specific strengths that aggregates hide. A secondary analysis, however, finds Cronbach's alpha = 0.98 across nominally distinct dimensions and rank correlations >= 0.93 with the total for eight of nine: a halo effect indicating that demonstration-based scoring measures presentation as much as capability. From these lessons we propose a gate-then-score evaluation protocol with four layers (capability, execution reliability, governance gates, business value), trace-level reliability metrics estimated over repeated runs, an autonomy-readiness scale and a reproducibility protocol. We state precisely what the present evidence supports, and what a controlled harness must add to turn enterprise platform evaluation from structured judgement into measurement.
BibTeX
@misc{ali2026capability,
  author = {Abbas Raza Ali and Muhammad Ajmal Siddiqui and Moona Zahid},
  title = {{A Capability Framework for Evaluating Agentic AI Platforms in Regulated Enterprises}},
  year = {2026},
  eprint = {2611.24039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24039},
}
Figure