RLHFless: Serverless Computing for Efficient RLHF

Rui Wei, Hanfei Yu, Shubham Jain, Yogarajan Sivakumar, Devesh Tiwari, Jian Li, Seung-Jong Park, Hao WangarXiv 2026 · arXiv

TL;DR

RLHFless is the first serverless computing framework designed for efficient, synchronous Reinforcement Learning from Human Feedback (RLHF) training. By dynamically scaling resources on demand, deduplicating key-value (KV) cache prefill computations, and grouping prompts by predicted response lengths, it eliminates the massive resource wastage inherent in traditional static server setups. The framework achieves up to a 1.35× training speedup and a 44.8% reduction in GPU infrastructure costs.

What Problem Does This Paper Address?

Post-training alignment via RLHF—essential for modern models like DeepSeek-R1—is incredibly resource-intensive. The training loop inherently alternates between an inference-heavy generation phase (sampling responses from the LLM) and a training-heavy learning phase (updating model weights based on rewards).

Figure 1: RLHF’s dataflow in one training step, including the generation phase, preparation phase, and learning phase. Some details, like the use of the critic model, can vary de- pending on the specific algorithm used.

Traditional frameworks rely on serverful architectures with static resource allocations. This introduces severe structural inefficiencies:

  1. Inter-component Idleness: GPUs allocated for generation sit completely idle during the learning phase, and vice versa.
  2. Intra-component Stragglers: LLM response lengths vary wildly. In a statically allocated cluster, faster-generating sequences finish early, leaving GPUs underutilized while waiting for a few "long-tail" responses to finish.
  3. Redundant Computation: Many RLHF algorithms generate multiple responses for the same prompt or share common prompt prefixes, leading to highly repetitive and wasteful KV cache prefill operations.
Figure 2: (a) RLHF’s staged workflow causes idle time be- tween components. (b) Idle components and (c) repeated cal- culation in RLHF lead to resource wastage.

Methods

RLHFless addresses these challenges by replacing static server allocations with an elastic, serverless architecture featuring four core components:

  • Deduplicated Prefill: Instead of every parallel worker computing the prefill phase independently, RLHFless uses a dedicated, compute-bound prefill actor to process unique prompt prefixes exactly once. The resulting KV cache is then shared across downstream decoding actors.
  • Dynamic Prompt Assignment: To eliminate stragglers, the framework predicts upcoming response lengths using an Exponentially Weighted Moving Average (EWMA) of historical data. It then groups prompts of similar expected lengths into the same decoding actors, allowing short-workload actors to terminate and release their serverless resources early.
  • Cut-and-Migrate Backoff: To handle stochastic variations where a prompt's response length is underestimated, RLHFless implements a runtime correction. If an actor is delayed by a long-running response, it cuts the generation at a predefined threshold and migrates the state to an available slot on another active actor.
  • Cost-Aware Scaling & Locality Placement: Before each step, a planning module solves an optimization problem to find the "sweet spot" number of actors that minimizes both execution time and financial cost. It co-locates the prefill actor with the learner and prioritizes placing heavy decoding workloads on physically closer GPU nodes to hide network transfer latencies.

Datasets

The authors evaluated RLHFless using standard, widely accepted benchmarks that exhibit diverse response length characteristics:

  • GSM8K: A dataset of high-school-level mathematical word problems, which typically trigger long, step-by-step reasoning chains.
  • GPQA: A challenging, graduate-level Google-proof science question dataset.
  • LiveCodeBench: A holistic benchmark for evaluating LLMs on complex coding tasks.

Large-scale scalability simulations were also conducted using synthetic workloads modeled after Llama2-70B training traces.

Key Insights and Ideas

  • The Predictability of RLHF Workloads: Unlike online LLM serving where user queries are entirely unpredictable, RLHF training involves repeated sampling over a fixed dataset across multiple epochs. The authors realized that response lengths for specific prompts follow stable trends over training steps. This allows simple, model-free historical tracking (like EWMA) to act as an accurate workload predictor, avoiding the need for expensive auxiliary machine learning models.
  • Serverless is a Natural Fit for Staged Pipelines: Because synchronous RLHF requires strict dependencies between sampling and learning, the "pay-as-you-go" granularity of serverless functions perfectly monetizes the idle time that plagues serverful setups. Spinning down generation actors the moment they finish directly cuts costs.
  • Exploiting Length Diversity: Instead of treating variable response lengths as an scheduling nuisance, RLHFless turns it into an advantage. By sorting and grouping prompts by length, the system can systematically release blocks of GPUs early.
Figure 4: Dynamic resource demands caused by: (a) signif- icantly varying response lengths across and within differ- ent types of datasets, such as mathematical problems [36], general scientific questions [50], and coding tasks [28]; (b) changing response lengths during RLHF training.

Key Results

  • Overall Efficiency: Compared to the state-of-the-art serverful baseline (VERL), RLHFless achieves up to a 1.35× speedup and up to a 44.8% cost reduction on physical testbeds (AWS instances with NVIDIA L40S GPUs).
  • Algorithm-Specific Gains: The framework delivers higher efficiency gains on Group Relative Policy Optimization (GRPO) than on Proximal Policy Optimization (PPO). This is because GRPO is highly sampling-heavy (generating many responses per prompt), leaving more room for KV cache deduplication and elastic scaling to eliminate waste.
  • Scalability: In large-scale simulated environments scaling up to 20 nodes (160 NVIDIA H100 GPUs) running Llama2-70B, RLHFless maintained a stable scheduling overhead of under 30 ms while providing a 1.23× speedup and 38.7% cost reduction.

Limitations and Future Work

  • Synchronous Focus: RLHFless is explicitly optimized for synchronous RLHF training. While its prompt assignment and actor-scaling strategies are conceptually orthogonal to asynchronous or off-policy RLHF frameworks, adapting them to handle stale data and continuous streaming updates remains future work.
  • Hyperparameter Sensitivity: The framework relies on a few key hyperparameters, such as the cut-and-migrate threshold ($\tau$) and the actor-scaling weight ($\lambda$). While the paper provides robust default values ($\tau=0.7, \lambda=0.7$), these parameters may require initial profiling to find optimal trade-offs on highly customized hardware topologies.

Summary updated

Abstract

Reinforcement Learning from Human Feedback (RLHF) has been widely applied to Large Language Model (LLM) post-training to align model outputs with human preferences. Recent models, such as DeepSeek-R1, have also shown RLHF's potential to improve LLM reasoning on complex tasks. In RL, inference and training co-exist, creating dynamic resource demands throughout the workflow. Compared to traditional RL, RLHF further challenges training efficiency due to expanding model sizes and resource consumption. Several RLHF frameworks aim to balance flexible abstraction and efficient execution. However, they rely on serverful infrastructures, which struggle with fine-grained resource variability. As a result, during synchronous RLHF training, idle time between or within RL components often causes overhead and resource wastage. To address these issues, we present RLHFless, the first scalable training framework for synchronous RLHF, built on serverless computing environments. RLHFless adapts to dynamic resource demands throughout the RLHF pipeline, pre-computes shared prefixes to avoid repeated computation, and uses a cost-aware actor scaling strategy that accounts for response length variation to find sweet spots with lower cost and higher speed. In addition, RLHFless assigns workloads efficiently to reduce intra-function imbalance and idle time. Experiments on both physical testbeds and a large-scale simulated cluster show that RLHFless achieves up to 1.35x speedup and 44.8% cost reduction compared to the state-of-the-art baseline.
BibTeX
@misc{wei2026rlhfless,
  author = {Rui Wei and Hanfei Yu and Shubham Jain and Yogarajan Sivakumar and Devesh Tiwari and Jian Li and Seung-Jong Park and Hao Wang},
  title = {{RLHFless: Serverless Computing for Efficient RLHF}},
  year = {2026},
  eprint = {2602.22718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.22718},
}

Loading related content...

Figure