What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays

Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li, Jianbing NiarXiv 2026 · arXiv

TL;DR

Third-party LLM API relays often promise high-end models like GPT-6 at a fraction of the official cost, but this study reveals significant hidden risks. By auditing 20 relay services, researchers found widespread model behavior discrepancies, ignored tool permissions, and inflated token billing that directly hits the user's wallet.

What Problem Does This Paper Address?

LLM API relays act as middlemen, sitting between a client application and model providers like OpenAI or Anthropic. Users flock to them for lower prices, regional access, or simpler payment methods. However, because the relay handles the plaintext request, it can silently modify instructions, substitute the requested model for a cheaper one, or misreport usage.

While previous research has looked at model "fingerprinting" to detect substitution, this paper argues that model identity is only one part of the service contract. A relay might provide the correct model but still fail by injecting unwanted content, ignoring security constraints (like disabling tools), or overcharging for tokens. The authors set out to measure these client-visible properties across multiple services and over time.

Methods

The researchers evaluated 20 relays offering access to GPT-6 Astra and Claude Opus 5. Their methodology focused on three pillars:

  1. Behavioral Auditing: Using "Knowledge Boundary as Fingerprint" (KBF), they designed questions that the official models answer correctly but cheaper substitutes (like Claude Haiku 4.5 or GPT-6 Luna) typically miss.
  2. Permission & Content Checks: They sent requests with tools explicitly disabled (tool_choice: none) to see if relays respected the restriction. They also screened over 3,000 responses for potential code injections or malicious content.
  3. Usage & Billing Reconciliation: They sent identical requests to both the relay and the official API. By comparing the reported input token counts and checking account balance deductions, they could see if users were being billed for "phantom" tokens.

The study was conducted in two rounds, 11–18 days apart, to see if these behaviors were persistent or transient.

Measurement setup and comparisons of relay answers, tool permissions, and billed usage.

Datasets

  • Question Bank: 96 candidate questions across three categories: recent-event knowledge, parameter-scale-sensitive tasks, and capability-edge tasks.
  • Target Models: GPT-6 Astra and Claude Opus 5.
  • Substitute Models: Claude Haiku 4.5, GPT-6 Luna, and GLM-4.5-Flash (used to calibrate the detection tests).
  • Response Corpus: 3,332 screened responses used for the content injection audit.

Key Insights and Ideas

  • Price is a (Noisy) Signal: Low-price relays were significantly more likely to show behavioral differences compared to official models. However, paying "full price" didn't guarantee reliability; some same-price relays also failed behavioral tests.
  • The "Phantom Token" Tax: Several relays reported significantly more input tokens than the official API for the exact same request—in one case, a surplus of 60,000 tokens. Crucially, these extra tokens were not just reporting errors; they were actively billed to the user's account.
  • Permission Slippage: Four relays returned tool calls even when the user explicitly disabled them. This suggests the relay might be injecting its own system prompts or wrappers that override user settings, creating a potential security hole.
  • Reliability is Moving Target: Relay behavior is not static. Some services that passed the first round failed the second, and vice versa. This highlights the need for continuous monitoring rather than one-off evaluations.

Key Results

  • Accuracy Gaps: In the first round, 7 out of 10 low-price relays were flagged for behavioral differences from the official reference.
  • Billing Discrepancies: Five relays reported between 4,200 and 60,000 extra input tokens compared to official counts. For all services where balances could be checked, the account deduction matched the reported (inflated) usage to the cent.
  • Tool Violations: Despite explicit suppression, researchers observed 30 instances of relays returning tool calls in the first round.
  • Temporal Shifts: Retesting showed that 7 out of 35 relay-model pairs changed their status (gaining or losing a "flag") within just two weeks.
Proportion of relays with behavioral differences by price group.

Limitations and Future Work

The authors acknowledge that their methods can detect differences but cannot definitively identify the cause. A behavioral gap might be due to model substitution, but it could also be caused by heavy system prompts, different sampling settings, or relay-side caching. Furthermore, the study did not test for long-term conversation retention or credential theft.

Future work could involve developing Verifiable Relay Services. This would require infrastructure where providers sign their responses and usage counts, allowing users to cryptographically verify that the model they paid for is the one they actually received.

Summary updated

Abstract

Third-party application programming interface (API) relays offer access to large language models (LLMs), often at lower prices. Users depend on these services for model quality, instruction compliance, and accurate billing. However, existing studies leave unclear whether relays meet these expectations together. We test 20 relays offering GPT-6 Astra and Claude Opus 5. First, we compare answers to knowledge questions with batch-matched official responses. In our first-round sample, the analysis flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences. On retesting 35 pairs, seven gain or lose a flag under the original rules, and six under matched correction families. Because answer comparisons cannot reveal every delivery problem, we examine returned content and tool permissions. No injection is confirmed in 3,332 screened responses to a fixed, non-adversarial code and setup workload, yet four relays return tool calls despite instructions disabling them in the first round. Finally, to connect reported usage to actual costs, we compare input-token counts with official references and check balances. For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available (six services), charges follow reported usage at listed prices, including extra reported input tokens at one service. Our study shows that advertised models and prices are insufficient to judge relay reliability. It identifies unexplained billed token quantities and tool-permission violations as risks and shows why service evaluations need repeated measurements.
BibTeX
@misc{wu2026what,
  author = {Xiaodong Wu and Zhimin Zhao and Qi Li and Chaklam Cheong and Xiangman Li and Jianbing Ni},
  title = {{What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays}},
  year = {2026},
  eprint = {2611.07650},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07650},
}
Figure