TL;DR
This paper introduces three staged optimizations for the vLLM Semantic Router that deliver a massive 98× reduction in end-to-end routing latency while keeping the GPU memory footprint under 800 MB. By combining a custom Flash Attention operator for AMD ROCm, zero-neural-overhead prompt compression, and near-streaming body processing, the router can safely co-locate on the same GPU as the primary LLM serving instance, completely eliminating the need for an expensive dedicated routing GPU.
What Problem Does This Paper Address?
Production Large Language Model (LLM) deployments increasingly rely on system-level routers to intercept incoming requests for safety filtering, jailbreak detection, PII masking, and intent-based routing before they reach the backend models. However, these routing layers introduce a strict latency penalty that directly inflates the user-perceived Time-to-First-Token (TTFT).
As real-world prompts grow to 8K–32K tokens due to Retrieval-Augmented Generation (RAG) and document analysis, standard Scaled Dot-Product Attention (SDPA) suffers from quadratic $O(n^2)$ memory scaling. When trying to run these routing classifiers on the same GPU as the main LLM serving instance (like vLLM), the router quickly runs out of memory (OOMs) because the LLM already claims almost all the High Bandwidth Memory (HBM). Offloading routing to CPUs avoids the OOMs but introduces unacceptable multi-second delays, while standard proxy buffering adds heavy JSON serialization overhead.
Methods
The authors address these bottlenecks using a three-stage pipeline:
- CK Flash Attention for ONNX Runtime on ROCm: While NVIDIA GPUs enjoy native Flash Attention support in ONNX Runtime, AMD ROCm support was missing, forcing a fallback to memory-heavy SDPA. The authors developed a custom operator and graph rewriter that replaces standard attention subgraphs with AMD's Composable Kernel (CK) Flash Attention, dropping memory complexity to $O(n)$ and mapping ModernBERT’s alternating local and global attention layers to CK's sliding-window parameters.
- Neural-Inference-Free Prompt Compression: Instead of using a secondary, smaller LLM to compress prompts (which adds massive latency), the authors built a fast, classical NLP pipeline on the CPU. It combines TextRank (centrality), U-shaped position weighting (prioritizing the beginning and end of prompts), TF-IDF, and novelty scoring to compress long contexts down to a fixed ~512 tokens before classification.
- Near-Streaming Body Processing: Utilizing an adaptive chunked handler for Envoy, the router inspects the first kilobyte of a request. If a specific model is requested, it switches to a zero-copy passthrough mode. If classification is needed ("auto" mode), it incrementally splits sentences and computes vectors as chunks arrive, overlapping I/O with preprocessing.
Datasets
The system was evaluated using two primary approaches:
- Synthetic Benchmarks: Generated at target token lengths (500 to 16K tokens) containing embedded jailbreak prefixes, PII tokens, and domain-specific content to force all classifiers to fire simultaneously.
- Real Content Evaluation Corpus: 384 test cases constructed from 8 full Wikipedia articles across 8 domains, tested at 4 prompt lengths (2K to 16K tokens) with 12 signal-position combinations to evaluate classification accuracy.
Key Insights and Ideas
- Compression as a Denoising Step: Surprisingly, compressing the prompt improves domain classification accuracy (from 53.1% to 61.2%) and PII detection (from 78.5% to 92.4%). Long documents contain narrative filler that dilutes a classifier's attention; stripping this noise concentrates the signal into the classifier's effective attention span, mitigating the "lost in the middle" effect.
- Classical NLP is Not Dead: While recent trends favor neural prompt compressors (like LLMLingua), they are too slow for real-time routing. Classical, unsupervised algorithms can achieve the necessary compression in milliseconds rather than seconds.
- Smart Co-location Saves Real Money: By capping the input to 512 tokens via compression, the router's memory footprint becomes a small, predictable constant (<800 MB), making it perfectly safe to share a GPU with vLLM and saving thousands of dollars per node in cluster infrastructure costs.
Key Results
- 98× Latency Reduction: At 8K tokens, end-to-end routing latency plummeted from 4,918 ms (CPU baseline) to just 50 ms.
- OOM Elimination: Flash Attention enabled the router to handle up to 32K tokens and 20 concurrent requests where standard attention completely OOMs.
- High Throughput: The optimized router sustains ~60 requests/second at 16K tokens under concurrent load (10 concurrent requests).
Limitations and Future Work
- Heuristic Tokenization: The prompt compression currently relies on a character-length heuristic rather than the exact model tokenizer, which can lead to less precise token budget control.
- Memory Buffering: In "accumulate" mode, the streaming handler still buffers the entire request body in memory before classification, meaning the savings are primarily in reduced JSON processing time rather than total system memory reduction.
Summary updated