Abstract
Large Language Model (LLM) inference has emerged as a fundamental paradigm. In real-world scenarios, variations in output length cause severe workload imbalance in the decode phase, particularly for long-output reasoning tasks. Existing systems, such as PD disaggregation architectures, rely on static prefill-to-decode scheduling, which often results in SLO violations and OOM failures under evolving decode workloads. In this paper, we propose ARES, an adaptive decoding rescheduling system powered by length prediction to anticipate future workloads. Our core contributions include: (1) A lightweight and continuous LLM-native prediction method that leverages LLM hidden state to model remaining generation length with high precision (reducing MAE by 49.42%) and low overhead (cutting predictor parameters by 93.28%); (2) A rescheduling solution in decode phase with: A dynamic balancing mechanism that integrates current and predicted workloads, reducing P99 TPOT by 74.77% and achieving up to 2.24 times higher goodput.
Introduction
In modern Large Language Model (LLM) serving, the inference process is typically split into two stages: Prefill (processing the input prompt) and Decode (generating tokens autoregressively). State-of-the-art systems utilize "Prefill-Decode (PD) Disaggregation," where different GPUs are dedicated to each stage to optimize for their distinct hardware requirements.
However, a significant challenge remains in the Decode phase. Output lengths are highly unpredictable, especially with reasoning tasks or "Chain-of-Thought" prompts. When a request finishes prefill, it is assigned to a decode server. Static assignment often leads to workload imbalance: one server might be overwhelmed by several "heavy" requests generating thousands of tokens, while another sits idle. This imbalance causes high "Time Per Output Token" (TPOT) and potential Out-of-Memory (OOM) errors as Key-Value (KV) caches grow beyond GPU capacity.
Methodology: The ARES System
ARES (Adaptive REscheduling System) addresses these challenges through a dynamic rescheduling approach. The system architecture integrates a novel predictor and a multi-stage scheduler to manage workloads proactively.
1. Lightweight & Continuous LLM-Native Predictor
ARES moves away from bulky auxiliary models by using the target LLM's own internal hidden states. By feeding the hidden state vector of the last token from the final transformer layer into a 4-layer MLP, ARES captures the model's "intent." This "LLM-native" approach is more accurate and efficient than previous methods. Furthermore, ARES re-predicts the remaining length every 20 tokens, allowing for continuous refinement as generation progresses.
2. Multi-Stage Rescheduling Strategy
The ARES scheduler periodically evaluates cluster health through three phases:
- Classification: Identifies "overloaded" and "underloaded" servers based on current and predicted token loads.
- Candidate Enumeration: Filters requests where migration overhead is justified by the remaining generation length.
- Optimal Selection: Simulates migrations to find the configuration that most effectively minimizes workload variance across the cluster.
KV caches are transferred asynchronously using NVIDIA’s Inference Transfer Library to ensure that other requests on the same server are not paused during the move.
Evaluation and Results
The system was evaluated using the ShareGPT and Alpaca datasets on a DeepSeek-R1-Distill-Qwen-7B model, which is known for long-form reasoning.
Performance Gains
- Goodput: ARES achieved up to 2.24x higher goodput (requests per second meeting latency requirements) compared to standard vLLM baselines.
- Latency: P99 TPOT was reduced by 74.77%, significantly improving the user experience for long-running requests.
- Prediction Efficiency: The LLM-native predictor reduced Mean Absolute Error (MAE) by 49.42% while using 93.28% fewer parameters than previous state-of-the-art auxiliary predictors.
Key Insights
The researchers found that a token's hidden state is significantly more informative for length prediction than its KV cache entry, as it encapsulates the model's trajectory. Additionally, they observed that execution time and memory usage correlate linearly with the number of batched tokens, allowing the use of "token load" as a unified scheduling metric.
Discussion and Limitations
While ARES significantly improves performance, it is sensitive to network bandwidth. Low-speed interconnects (below 10 Gbps) can make KV cache migration overhead exceed the scheduling benefits. Additionally, the small MLP predictor requires model-specific training, although this process is relatively fast (approx. 1 hour on a single GPU). Future work will explore interactions with quantization techniques and scaling to multi-node cluster environments.
Summary updated