98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin ChenarXiv 2026 · arXiv

TL;DR

This paper introduces three staged optimizations for the vLLM Semantic Router that deliver a massive 98× reduction in end-to-end routing latency while keeping the GPU memory footprint under 800 MB. By combining a custom Flash Attention operator for AMD ROCm, zero-neural-overhead prompt compression, and near-streaming body processing, the router can safely co-locate on the same GPU as the primary LLM serving instance, completely eliminating the need for an expensive dedicated routing GPU.

What Problem Does This Paper Address?

Production Large Language Model (LLM) deployments increasingly rely on system-level routers to intercept incoming requests for safety filtering, jailbreak detection, PII masking, and intent-based routing before they reach the backend models. However, these routing layers introduce a strict latency penalty that directly inflates the user-perceived Time-to-First-Token (TTFT).

As real-world prompts grow to 8K–32K tokens due to Retrieval-Augmented Generation (RAG) and document analysis, standard Scaled Dot-Product Attention (SDPA) suffers from quadratic $O(n^2)$ memory scaling. When trying to run these routing classifiers on the same GPU as the main LLM serving instance (like vLLM), the router quickly runs out of memory (OOMs) because the LLM already claims almost all the High Bandwidth Memory (HBM). Offloading routing to CPUs avoids the OOMs but introduces unacceptable multi-second delays, while standard proxy buffering adds heavy JSON serialization overhead.

Fig. 3. Attention memory per classifier session vs. sequence length. SDPA exceeds the ∼718 MB available (dashed line) at ∼6K tokens. FA stays below but grows linearly. With compression, all inputs are reduced to 512 tokens, yielding a constant footprint regardless of original prompt length.

Methods

The authors address these bottlenecks using a three-stage pipeline:

  1. CK Flash Attention for ONNX Runtime on ROCm: While NVIDIA GPUs enjoy native Flash Attention support in ONNX Runtime, AMD ROCm support was missing, forcing a fallback to memory-heavy SDPA. The authors developed a custom operator and graph rewriter that replaces standard attention subgraphs with AMD's Composable Kernel (CK) Flash Attention, dropping memory complexity to $O(n)$ and mapping ModernBERT’s alternating local and global attention layers to CK's sliding-window parameters.
  2. Neural-Inference-Free Prompt Compression: Instead of using a secondary, smaller LLM to compress prompts (which adds massive latency), the authors built a fast, classical NLP pipeline on the CPU. It combines TextRank (centrality), U-shaped position weighting (prioritizing the beginning and end of prompts), TF-IDF, and novelty scoring to compress long contexts down to a fixed ~512 tokens before classification.
  3. Near-Streaming Body Processing: Utilizing an adaptive chunked handler for Envoy, the router inspects the first kilobyte of a request. If a specific model is requested, it switches to a zero-copy passthrough mode. If classification is needed ("auto" mode), it incrementally splits sentences and computes vectors as chunks arrive, overlapping I/O with preprocessing.
Fig. 2. Near-streaming body handler state machine. The first chunk determines the path: specified models bypass body inspection entirely; “auto” requests accumulate for classification with incremental preprocessing.

Datasets

The system was evaluated using two primary approaches:

  • Synthetic Benchmarks: Generated at target token lengths (500 to 16K tokens) containing embedded jailbreak prefixes, PII tokens, and domain-specific content to force all classifiers to fire simultaneously.
  • Real Content Evaluation Corpus: 384 test cases constructed from 8 full Wikipedia articles across 8 domains, tested at 4 prompt lengths (2K to 16K tokens) with 12 signal-position combinations to evaluate classification accuracy.

Key Insights and Ideas

  • Compression as a Denoising Step: Surprisingly, compressing the prompt improves domain classification accuracy (from 53.1% to 61.2%) and PII detection (from 78.5% to 92.4%). Long documents contain narrative filler that dilutes a classifier's attention; stripping this noise concentrates the signal into the classifier's effective attention span, mitigating the "lost in the middle" effect.
  • Classical NLP is Not Dead: While recent trends favor neural prompt compressors (like LLMLingua), they are too slow for real-time routing. Classical, unsupervised algorithms can achieve the necessary compression in milliseconds rather than seconds.
  • Smart Co-location Saves Real Money: By capping the input to 512 tokens via compression, the router's memory footprint becomes a small, predictable constant (<800 MB), making it perfectly safe to share a GPU with vLLM and saving thousands of dollars per node in cluster infrastructure costs.
Co-located deployment on MI300X GPU.

Key Results

  • 98× Latency Reduction: At 8K tokens, end-to-end routing latency plummeted from 4,918 ms (CPU baseline) to just 50 ms.
  • OOM Elimination: Flash Attention enabled the router to handle up to 32K tokens and 20 concurrent requests where standard attention completely OOMs.
  • High Throughput: The optimized router sustains ~60 requests/second at 16K tokens under concurrent load (10 concurrent requests).
Fig. 4. E2E latency comparison (log scale). GPU+FA with streaming and compression achieves 50 ms at 8K tokens. At 16K+, only GPU+FA operates (108 ms at 16K; Table VIII).

Limitations and Future Work

  • Heuristic Tokenization: The prompt compression currently relies on a character-length heuristic rather than the exact model tokenizer, which can lead to less precise token budget control.
  • Memory Buffering: In "accumulate" mode, the streaming handler still buffers the entire request body in memory before classification, meaning the savings are primarily in reduced JSON processing time rather than total system memory reduction.

Summary updated

Abstract

System-level routers that intercept LLM requests for safety classification, domain routing, and PII detection must be both fast and operationally lightweight: they should add minimal latency to every request, yet not require a dedicated GPU -- an expensive resource better used for LLM inference itself. When the router co-locates on the same GPU as vLLM serving instances, standard attention's $O(n^2)$ memory makes long-context classification (8K--32K tokens) impossible: at 8K tokens, three concurrent classifiers need ${\sim}$4.5\,GB for attention masks alone, far exceeding the memory left by vLLM. We present three staged optimizations for the vLLM Semantic Router, benchmarked on AMD Instinct MI300X, that solve both the latency and the memory problem. \emph{Stage~1}: a custom CK Flash Attention operator for ONNX Runtime on ROCm reduces attention memory from $O(n^2)$ to $O(n)$ and end-to-end (E2E) latency from 4{,}918\,ms to 127\,ms (\textbf{38.7$\times$}), enabling 8K--32K tokens where SDPA OOMs. \emph{Stage~2}: classical NLP prompt compression (TextRank, position weighting, TF-IDF, and novelty scoring) reduces all inputs to ${\sim}$512 tokens without neural inference, capping both latency and GPU memory at a constant regardless of original prompt length (E2E 127$\to$62\,ms, \textbf{2.0$\times$}). \emph{Stage~3}: near-streaming body processing with adaptive chunking and zero-copy JSON eliminates serialization overhead (E2E 62$\to$50\,ms, \textbf{1.2$\times$}). Cumulatively: \textbf{98$\times$} improvement (4{,}918\,ms to 50\,ms), 16K-token routing in 108\,ms, and a total router GPU footprint under 800\,MB -- small enough to share a GPU with LLM serving and removing the need for a dedicated accelerator. Stage~1 targets AMD ROCm (NVIDIA GPUs already have FlashAttention via cuDNN); Stages~2 and~3 are hardware-agnostic.
BibTeX
@misc{liu2026times,
  author = {Xunzhuo Liu and Bowei He and Xue Liu and Andy Luo and Haichen Zhang and Huamin Chen},
  title = {{98$\textbackslash\{\}times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router}},
  year = {2026},
  eprint = {2603.12646},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.12646},
}
Figure