TL;DR
Third-party LLM API relays often promise high-end models like GPT-6 at a fraction of the official cost, but this study reveals significant hidden risks. By auditing 20 relay services, researchers found widespread model behavior discrepancies, ignored tool permissions, and inflated token billing that directly hits the user's wallet.
What Problem Does This Paper Address?
LLM API relays act as middlemen, sitting between a client application and model providers like OpenAI or Anthropic. Users flock to them for lower prices, regional access, or simpler payment methods. However, because the relay handles the plaintext request, it can silently modify instructions, substitute the requested model for a cheaper one, or misreport usage.
While previous research has looked at model "fingerprinting" to detect substitution, this paper argues that model identity is only one part of the service contract. A relay might provide the correct model but still fail by injecting unwanted content, ignoring security constraints (like disabling tools), or overcharging for tokens. The authors set out to measure these client-visible properties across multiple services and over time.
Methods
The researchers evaluated 20 relays offering access to GPT-6 Astra and Claude Opus 5. Their methodology focused on three pillars:
- Behavioral Auditing: Using "Knowledge Boundary as Fingerprint" (KBF), they designed questions that the official models answer correctly but cheaper substitutes (like Claude Haiku 4.5 or GPT-6 Luna) typically miss.
- Permission & Content Checks: They sent requests with tools explicitly disabled (
tool_choice: none) to see if relays respected the restriction. They also screened over 3,000 responses for potential code injections or malicious content. - Usage & Billing Reconciliation: They sent identical requests to both the relay and the official API. By comparing the reported input token counts and checking account balance deductions, they could see if users were being billed for "phantom" tokens.
The study was conducted in two rounds, 11–18 days apart, to see if these behaviors were persistent or transient.
Datasets
- Question Bank: 96 candidate questions across three categories: recent-event knowledge, parameter-scale-sensitive tasks, and capability-edge tasks.
- Target Models: GPT-6 Astra and Claude Opus 5.
- Substitute Models: Claude Haiku 4.5, GPT-6 Luna, and GLM-4.5-Flash (used to calibrate the detection tests).
- Response Corpus: 3,332 screened responses used for the content injection audit.
Key Insights and Ideas
- Price is a (Noisy) Signal: Low-price relays were significantly more likely to show behavioral differences compared to official models. However, paying "full price" didn't guarantee reliability; some same-price relays also failed behavioral tests.
- The "Phantom Token" Tax: Several relays reported significantly more input tokens than the official API for the exact same request—in one case, a surplus of 60,000 tokens. Crucially, these extra tokens were not just reporting errors; they were actively billed to the user's account.
- Permission Slippage: Four relays returned tool calls even when the user explicitly disabled them. This suggests the relay might be injecting its own system prompts or wrappers that override user settings, creating a potential security hole.
- Reliability is Moving Target: Relay behavior is not static. Some services that passed the first round failed the second, and vice versa. This highlights the need for continuous monitoring rather than one-off evaluations.
Key Results
- Accuracy Gaps: In the first round, 7 out of 10 low-price relays were flagged for behavioral differences from the official reference.
- Billing Discrepancies: Five relays reported between 4,200 and 60,000 extra input tokens compared to official counts. For all services where balances could be checked, the account deduction matched the reported (inflated) usage to the cent.
- Tool Violations: Despite explicit suppression, researchers observed 30 instances of relays returning tool calls in the first round.
- Temporal Shifts: Retesting showed that 7 out of 35 relay-model pairs changed their status (gaining or losing a "flag") within just two weeks.
Limitations and Future Work
The authors acknowledge that their methods can detect differences but cannot definitively identify the cause. A behavioral gap might be due to model substitution, but it could also be caused by heavy system prompts, different sampling settings, or relay-side caching. Furthermore, the study did not test for long-term conversation retention or credential theft.
Future work could involve developing Verifiable Relay Services. This would require infrastructure where providers sign their responses and usage counts, allowing users to cryptographically verify that the model they paid for is the one they actually received.
Summary updated