Adaptive Rescheduling in Prefill-Decode Disaggregated LLM Inference

Zhibin Wang, Zetao Hong, Xue Li, Zhengliang Wang, Shipeng Li, Qifu Meng, Qing Wang, Chengying Huan, Rong Gu, Sheng Zhong, Chen Tian

Abstract

Large Language Model (LLM) inference has emerged as a fundamental paradigm. In real-world scenarios, variations in output length cause severe workload imbalance in the decode phase, particularly for long-output reasoning tasks. Existing systems, such as PD disaggregation architectures, rely on static prefill-to-decode scheduling, which often results in SLO violations and OOM failures under evolving decode workloads. In this paper, we propose ARES, an adaptive decoding rescheduling system powered by length prediction to anticipate future workloads. Our core contributions include: (1) A lightweight and continuous LLM-native prediction method that leverages LLM hidden state to model remaining generation length with high precision (reducing MAE by 49.42%) and low overhead (cutting predictor parameters by 93.28%); (2) A rescheduling solution in decode phase with: A dynamic balancing mechanism that integrates current and predicted workloads, reducing P99 TPOT by 74.77% and achieving up to 2.24 times higher goodput.

Introduction

In modern Large Language Model (LLM) serving, the inference process is typically split into two stages: Prefill (processing the input prompt) and Decode (generating tokens autoregressively). State-of-the-art systems utilize "Prefill-Decode (PD) Disaggregation," where different GPUs are dedicated to each stage to optimize for their distinct hardware requirements.

However, a significant challenge remains in the Decode phase. Output lengths are highly unpredictable, especially with reasoning tasks or "Chain-of-Thought" prompts. When a request finishes prefill, it is assigned to a decode server. Static assignment often leads to workload imbalance: one server might be overwhelmed by several "heavy" requests generating thousands of tokens, while another sits idle. This imbalance causes high "Time Per Output Token" (TPOT) and potential Out-of-Memory (OOM) errors as Key-Value (KV) caches grow beyond GPU capacity.

Workload imbalance in LLM decoding due to variable output lengths.

Methodology: The ARES System

ARES (Adaptive REscheduling System) addresses these challenges through a dynamic rescheduling approach. The system architecture integrates a novel predictor and a multi-stage scheduler to manage workloads proactively.

The ARES system architecture featuring the LLM-native length predictor and migration scheduler.

1. Lightweight & Continuous LLM-Native Predictor

ARES moves away from bulky auxiliary models by using the target LLM's own internal hidden states. By feeding the hidden state vector of the last token from the final transformer layer into a 4-layer MLP, ARES captures the model's "intent." This "LLM-native" approach is more accurate and efficient than previous methods. Furthermore, ARES re-predicts the remaining length every 20 tokens, allowing for continuous refinement as generation progresses.

2. Multi-Stage Rescheduling Strategy

The ARES scheduler periodically evaluates cluster health through three phases:

  • Classification: Identifies "overloaded" and "underloaded" servers based on current and predicted token loads.
  • Candidate Enumeration: Filters requests where migration overhead is justified by the remaining generation length.
  • Optimal Selection: Simulates migrations to find the configuration that most effectively minimizes workload variance across the cluster.

KV caches are transferred asynchronously using NVIDIA’s Inference Transfer Library to ensure that other requests on the same server are not paused during the move.

Evaluation and Results

The system was evaluated using the ShareGPT and Alpaca datasets on a DeepSeek-R1-Distill-Qwen-7B model, which is known for long-form reasoning.

Performance Gains

  • Goodput: ARES achieved up to 2.24x higher goodput (requests per second meeting latency requirements) compared to standard vLLM baselines.
  • Latency: P99 TPOT was reduced by 74.77%, significantly improving the user experience for long-running requests.
  • Prediction Efficiency: The LLM-native predictor reduced Mean Absolute Error (MAE) by 49.42% while using 93.28% fewer parameters than previous state-of-the-art auxiliary predictors.

Key Insights

The researchers found that a token's hidden state is significantly more informative for length prediction than its KV cache entry, as it encapsulates the model's trajectory. Additionally, they observed that execution time and memory usage correlate linearly with the number of batched tokens, allowing the use of "token load" as a unified scheduling metric.

Discussion and Limitations

While ARES significantly improves performance, it is sensitive to network bandwidth. Low-speed interconnects (below 10 Gbps) can make KV cache migration overhead exceed the scheduling benefits. Additionally, the small MLP predictor requires model-specific training, although this process is relatively fast (approx. 1 hour on a single GPU). Future work will explore interactions with quantization techniques and scaling to multi-node cluster environments.

Summary updated

Abstract

Large Language Model (LLM) inference has emerged as a fundamental paradigm. In real-world scenarios, variations in output length cause severe workload imbalance in the decode phase, particularly for long-output reasoning tasks. Existing systems, such as PD disaggregation architectures, rely on static prefill-to-decode scheduling, which often results in SLO violations and OOM failures under evolving decode workloads. In this paper, we propose ARES, an adaptive decoding rescheduling system powered by length prediction to anticipate future workloads. Our core contributions include: (1) A lightweight and continuous LLM-native prediction method that leverages LLM hidden state to model remaining generation length with high precision (reducing MAE by 49.42%) and low overhead (cutting predictor parameters by 93.28%); (2) A rescheduling solution in decode phase with : A dynamic balancing mechanism that integrates current and predicted workloads, reducing P99 TPOT by 74.77% and achieving up to 2.24 times higher goodput.
BibTeX
@inproceedings{wang2026adaptive,
  author = {Zhibin Wang and Zetao Hong and Xue Li and Zhengliang Wang and Shipeng Li and Qifu Meng and Qing Wang and Chengying Huan and Rong Gu and Sheng Zhong and Chen Tian},
  title = {{Adaptive Rescheduling in Prefill-Decode Disaggregated LLM Inference}},
  booktitle = {HPDC},
  year = {2026},
  doi = {10.1145/3806645.3807813},
}
Figure