TL;DR
PayPal researchers present Cosmos, a high-performance LLM gateway built on Envoy that centralizes governance, cost control, and API translation for enterprise-scale AI. By splitting work between an out-of-process control plane and an in-process Rust WebAssembly filter, the system handles 200 billion tokens per day with a median latency overhead of just 4 ms.
What Problem Does This Paper Address?
As large organizations rush to integrate Generative AI, they face a "wild west" of implementation. Different teams use different vendors (OpenAI, Anthropic, Google), leading to four major failure modes:
- Credential Sprawl: Long-lived API keys scattered across applications with no central rotation or attribution.
- Format Lock-in: Applications become tightly coupled to a specific vendor's API dialect, making model switching or failover an engineering nightmare.
- Ungoverned Spend: Lack of organization-aware budgets or early warnings for runaway costs.
- Inconsistent Security: Uneven data protection (DLP) and visibility across different integrations.
The authors argue that direct application-to-vendor integration does not survive at the scale of thousands of engineers and hundreds of production workloads.
Methods
The core of the solution is a two-tier Envoy architecture designed to be "strict and fast at the same time."
- Latency-Profile Splitting: The gateway separates I/O-bound governance decisions (authentication, budget checks) from CPU-bound payload work (translation, metering).
- Control Path (Out-of-Process): A Go-based service handles complex logic like identity validation and Open Policy Agent (OPA) evaluations.
- Data Path (In-Process): A Rust-based WebAssembly (Wasm) filter performs wire-format translation and token metering chunk-by-chunk. This ensures that streamed response chunks never wait on a network call to an external service.
- Canonical Intermediate Representation (IR): Instead of writing $N \times M$ converters for every possible API pair, the gateway translates all incoming traffic into a single internal format before encoding it for the upstream provider.
- Policy as Code: Governance is declarative, using OPA's Rego language. This allows for "what-if" queries and unit-testing of policies before they hit production.
- Data Loss Prevention (DLP): An inline layer uses a mix of pattern rules and purpose-trained models to block, redact, or pseudonymize sensitive data before it leaves the enterprise boundary.
Datasets
The paper evaluates the system using two primary sources:
- Production Traffic: Historical data from PayPal’s internal deployment, serving 12,000+ developers and hundreds of use cases.
- Synthetic Benchmark: A performance test using a Locust load generator against a synthetic upstream service (fixed 110 ms latency) to isolate the gateway's overhead.
Key Insights and Ideas
- Scopes as Views, Not Storage: One of the most clever FinOps features is how budgets are handled. Spend is recorded against immutable model identifiers. Budget "scopes" are simply views over these counters. This means if a manager creates a new budget rule today, it can retroactively apply to spend already recorded in the current window without any data migration.
- Identity-Aware Hierarchy: Policies are anchored to the organizational hierarchy rather than static team lists. If an engineer moves to a new department, their AI access and budgets follow them automatically.
- Shadow-Mode Rollouts: New policies are always rolled out in "shadow mode" first—tracking utilization and logging rejections without actually blocking traffic—to ensure they don't accidentally take production systems offline.
- The "Strangler-Fig" Migration: To avoid a "big bang" migration, PayPal placed Envoy in front of existing services and moved capabilities (auth, spend, translation) into the gateway one by one.
Key Results
- Performance: In benchmarks, the gateway added a median overhead of 4 ms and a p99 of 11 ms at 2,200+ requests per second.
- Scale: As of late 2026, the system handles ~200 billion tokens per day across 100+ models with 99.999% availability.
- Real-Time Support: The gateway successfully supports 2,000+ concurrent real-time voice calls (WebSockets) with zero dropped calls during deployments.
- Cost Savings: Gateway-level prompt-cache shaping achieved hit rates of up to 96% on agentic coding workloads, significantly reducing vendor costs.
Limitations and Future Work
- Centralization Risk: By brokering credentials for every provider, the gateway becomes a high-value target. Its security and change control must meet the highest possible standards.
- DLP Accuracy: The authors acknowledge that DLP is probabilistic. While it reduces exposure, it does not eliminate it entirely and requires ongoing tuning.
- Future Directions: The team is working on automatic model fallbacks, extending the gateway to handle Model Context Protocol (MCP) tool calls for autonomous agents, and implementing semantic caching to further reduce costs and latency.
Summary updated