TL;DR
This paper introduces a progressively deployable, "incentive-compatible" cryptographic stack to secure the LLM router ecosystem. By using techniques like Format-Preserving Encryption (FPE) and attenuable capabilities, the authors demonstrate how to protect user credentials and prompt integrity from malicious or compromised routers with less than 5% latency overhead.
What Problem Does This Paper Address?
As LLM applications scale, developers rarely connect directly to a model provider (like OpenAI or Anthropic). Instead, they use API routers (e.g., LiteLLM) for load balancing, cost tracking, and failover.
The problem is that these routers act as "man-in-the-middle" entities. Because they terminate TLS connections to inspect traffic for routing logic, they see everything in plaintext: your prompts, your tool-call schemas, and—most dangerously—your embedded credentials (like AWS keys or database passwords). Recent audits have shown that some "shadow" routers are already actively draining user credentials.
Existing defenses like behavioral detection are insufficient because a clever router can behave perfectly during testing and only attack when a specific "hidden predicate" is met. Furthermore, previous cryptographic proposals often failed because they ignored the business incentives of routers (who want to keep their provider lists secret) and providers (who don't want legal liability for signed tool calls).
Methods
The authors propose a five-phase "trust-reduction stack" designed to be adopted incrementally. The key innovation is incentive alignment: each phase provides immediate value to at least one stakeholder without requiring a "flag day" where everyone must upgrade at once.
- Phase 0 (Client-side protection): The core implemented contribution. It uses Format-Preserving Encryption (FPE) to tokenize sensitive strings (like API keys) before they leave the client. Because the tokens look like the original data (e.g., a tokenized AWS key still looks like an AWS key), the LLM can still reason about the prompt, but the router only sees "ciphertext" that it cannot decrypt.
- Phase 1-4 (The Roadmap): These phases introduce Ring Signatures for anonymous router attestation (protecting router business secrets), OAuth DPoP for short-lived capabilities, and Chameleon Hashes to allow routers to make legitimate schema changes (like renaming a parameter) without breaking a provider's cryptographic signature.
Datasets
The authors evaluated their Phase 0 implementation using:
- The "Mine" Corpus: A dataset of 12 credential families (AWS, OpenAI, GitHub, etc.) used to test "Secret Exposure Rate."
- ShareGPT & HumanEval: Subsets of these datasets were used to inject secrets and test if tokenization degraded the LLM's ability to follow instructions or write code.
- Live API Evaluation: A 300-sample evaluation using
gpt-4o-minito measure real-world utility and "LLM-as-a-judge" scoring.
Key Insights and Ideas
- The "Red Lines" of Adoption: The authors identified four constraints that kill security proposals: forcing routers to reveal their upstream providers, requiring providers to sign execution-relevant content (liability), invasive framework changes, and >5% latency overhead.
- FPE vs. Redaction: Simply "redacting" a secret (replacing it with
[REDACTED]) often breaks LLM reasoning because the model loses context. FPE provides "value indistinguishability"—the model sees a unique token it can work with, but the router sees nothing useful. - Unilateral Deployment: Phase 0 is "individually rational." A security-conscious enterprise can deploy the Phase 0 middleware today to protect their data without needing any cooperation from the router or the model provider.
Key Results
The authors implemented their Phase 0 defense as a middleware for LiteLLM and found:
- Security: The Secret Exposure Rate (SER) dropped to 0.00 for all matched credential families. Malicious routers could no longer "scrape" keys from the traffic.
- Performance: The P95 latency overhead was between 2.9% and 4.8%, successfully staying under the "5% red line" for production deployment.
- Utility: Using an LLM-as-a-judge (GPT-4o), they found that tokenizing prompts resulted in no statistically significant loss in task quality, satisfying a 0.05 non-inferiority margin.
Limitations and Future Work
The paper acknowledges several boundaries:
- Natural Language Leakage: Phase 0 protects "structured" secrets (like keys) but does not yet solve the problem of a user typing a password directly into a conversational sentence.
- Metadata & Timing: The current stack does not mitigate traffic analysis or metadata leaks (e.g., a router inferring a secret's existence by its length).
- Phases 1-4 Implementation: While Phase 0 is fully implemented and tested, the later phases remain a "research agenda" that requires broader industry coordination.
- Adversarial Evasion: While FPE is robust, highly adaptive routers might attempt to "guess" secrets if the domain is very small (like a 5-digit zip code), requiring further domain-extension techniques.
Summary updated