Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy

Mithun Sasidharan, Srinivasan Manoharan, Rupesh Thakur, Abhinav Tk, Ashwin Hegde, Anand GautamarXiv 2026 · arXiv

TL;DR

PayPal researchers present Cosmos, a high-performance LLM gateway built on Envoy that centralizes governance, cost control, and API translation for enterprise-scale AI. By splitting work between an out-of-process control plane and an in-process Rust WebAssembly filter, the system handles 200 billion tokens per day with a median latency overhead of just 4 ms.

What Problem Does This Paper Address?

As large organizations rush to integrate Generative AI, they face a "wild west" of implementation. Different teams use different vendors (OpenAI, Anthropic, Google), leading to four major failure modes:

  1. Credential Sprawl: Long-lived API keys scattered across applications with no central rotation or attribution.
  2. Format Lock-in: Applications become tightly coupled to a specific vendor's API dialect, making model switching or failover an engineering nightmare.
  3. Ungoverned Spend: Lack of organization-aware budgets or early warnings for runaway costs.
  4. Inconsistent Security: Uneven data protection (DLP) and visibility across different integrations.

The authors argue that direct application-to-vendor integration does not survive at the scale of thousands of engineers and hundreds of production workloads.

Methods

The core of the solution is a two-tier Envoy architecture designed to be "strict and fast at the same time."

  • Latency-Profile Splitting: The gateway separates I/O-bound governance decisions (authentication, budget checks) from CPU-bound payload work (translation, metering).
  • Control Path (Out-of-Process): A Go-based service handles complex logic like identity validation and Open Policy Agent (OPA) evaluations.
  • Data Path (In-Process): A Rust-based WebAssembly (Wasm) filter performs wire-format translation and token metering chunk-by-chunk. This ensures that streamed response chunks never wait on a network call to an external service.
  • Canonical Intermediate Representation (IR): Instead of writing $N \times M$ converters for every possible API pair, the gateway translates all incoming traffic into a single internal format before encoding it for the upstream provider.
  • Policy as Code: Governance is declarative, using OPA's Rego language. This allows for "what-if" queries and unit-testing of policies before they hit production.
  • Data Loss Prevention (DLP): An inline layer uses a mix of pattern rules and purpose-trained models to block, redact, or pseudonymize sensitive data before it leaves the enterprise boundary.
Figure 1: Architecture. One governed entry point fronts enterprise LLM traffic. On the unified API, the control service makes I/O-bound decisions out of process and fails closed, and the Wasm filter translates and meters in process. Real-time sessions use a connection-oriented path tuned for long-lived streams. The control plane publishes versioned policy bundles that the data plane enforces on every request. The shaded region is PayPal’s private network: self-hosted open-weight models run inside it, and external providers are reached with brokered credentials.

Datasets

The paper evaluates the system using two primary sources:

  1. Production Traffic: Historical data from PayPal’s internal deployment, serving 12,000+ developers and hundreds of use cases.
  2. Synthetic Benchmark: A performance test using a Locust load generator against a synthetic upstream service (fixed 110 ms latency) to isolate the gateway's overhead.

Key Insights and Ideas

  • Scopes as Views, Not Storage: One of the most clever FinOps features is how budgets are handled. Spend is recorded against immutable model identifiers. Budget "scopes" are simply views over these counters. This means if a manager creates a new budget rule today, it can retroactively apply to spend already recorded in the current window without any data migration.
  • Identity-Aware Hierarchy: Policies are anchored to the organizational hierarchy rather than static team lists. If an engineer moves to a new department, their AI access and budgets follow them automatically.
  • Shadow-Mode Rollouts: New policies are always rolled out in "shadow mode" first—tracking utilization and logging rejections without actually blocking traffic—to ensure they don't accidentally take production systems offline.
  • The "Strangler-Fig" Migration: To avoid a "big bang" migration, PayPal placed Envoy in front of existing services and moved capabilities (auth, spend, translation) into the gateway one by one.
Figure 2: Lifecycle of a unified request. Routing decisions happen in the body phase because the model is in the body. Streamed response bytes pass through only the in-process filter on the critical path.

Key Results

  • Performance: In benchmarks, the gateway added a median overhead of 4 ms and a p99 of 11 ms at 2,200+ requests per second.
  • Scale: As of late 2026, the system handles ~200 billion tokens per day across 100+ models with 99.999% availability.
  • Real-Time Support: The gateway successfully supports 2,000+ concurrent real-time voice calls (WebSockets) with zero dropped calls during deployments.
  • Cost Savings: Gateway-level prompt-cache shaping achieved hit rates of up to 96% on agentic coding workloads, significantly reducing vendor costs.

Limitations and Future Work

  • Centralization Risk: By brokering credentials for every provider, the gateway becomes a high-value target. Its security and change control must meet the highest possible standards.
  • DLP Accuracy: The authors acknowledge that DLP is probabilistic. While it reduces exposure, it does not eliminate it entirely and requires ongoing tuning.
  • Future Directions: The team is working on automatic model fallbacks, extending the gateway to handle Model Context Protocol (MCP) tool calls for autonomous agents, and implementing semantic caching to further reduce costs and latency.

Summary updated

Abstract

Large organizations consume large language models (LLMs) from several vendors at once, each with its own API dialect, credential model, streaming protocol and pricing. Integrated directly, this heterogeneity yields credential sprawl, format lock-in, unbounded spend and inconsistent governance. We present the design and production operation of the Cosmos LLM Gateway, the governed entry point for enterprise LLM traffic at PayPal. Built on Envoy, it splits work by latency profile: I/O-bound control decisions (authentication, credential brokering, policy and budget evaluation) run in an out-of-process external-processing service, while wire-format translation, token metering and prompt-cache shaping run in-proxy as a Rust WebAssembly filter, so that, by default, streamed response body chunks never wait on an out-of-process call. Models, providers and prices are largely validated data rather than code. Governance is declarative policy evaluated by Open Policy Agent and managed through a control plane with role-scoped authoring, approvals, pilots, shadow-mode rollout, usage notifications and versioned policy bundles. For use cases that require it, a data-loss-prevention layer can block, redact or pseudonymize detected sensitive content before it reaches a provider. Spend scopes are views over per-model usage counters, so a new budget applies to spend already recorded in its window without migration. As of September 2026, the gateway serves more than 12,000 developers and hundreds of business use cases, handling roughly 200 billion tokens per day across more than 100 models at 99.999% availability to business. In a benchmark of the core request path against a synthetic upstream, it adds a median of 4 ms and a p99 of 11 ms of overhead at more than 2,200 requests per second on two 4-vCPU instances. We report the architecture, policy model, evaluation and lessons learned.
BibTeX
@misc{sasidharan2026governing,
  author = {Mithun Sasidharan and Srinivasan Manoharan and Rupesh Thakur and Abhinav TK and Ashwin Hegde and Anand Gautam},
  title = {{Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy}},
  year = {2026},
  eprint = {2610.35509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35509},
}
Figure