TL;DR
RLHFless is the first serverless computing framework designed for efficient, synchronous Reinforcement Learning from Human Feedback (RLHF) training. By dynamically scaling resources on demand, deduplicating key-value (KV) cache prefill computations, and grouping prompts by predicted response lengths, it eliminates the massive resource wastage inherent in traditional static server setups. The framework achieves up to a 1.35× training speedup and a 44.8% reduction in GPU infrastructure costs.
What Problem Does This Paper Address?
Post-training alignment via RLHF—essential for modern models like DeepSeek-R1—is incredibly resource-intensive. The training loop inherently alternates between an inference-heavy generation phase (sampling responses from the LLM) and a training-heavy learning phase (updating model weights based on rewards).
Traditional frameworks rely on serverful architectures with static resource allocations. This introduces severe structural inefficiencies:
- Inter-component Idleness: GPUs allocated for generation sit completely idle during the learning phase, and vice versa.
- Intra-component Stragglers: LLM response lengths vary wildly. In a statically allocated cluster, faster-generating sequences finish early, leaving GPUs underutilized while waiting for a few "long-tail" responses to finish.
- Redundant Computation: Many RLHF algorithms generate multiple responses for the same prompt or share common prompt prefixes, leading to highly repetitive and wasteful KV cache prefill operations.
Methods
RLHFless addresses these challenges by replacing static server allocations with an elastic, serverless architecture featuring four core components:
- Deduplicated Prefill: Instead of every parallel worker computing the prefill phase independently, RLHFless uses a dedicated, compute-bound prefill actor to process unique prompt prefixes exactly once. The resulting KV cache is then shared across downstream decoding actors.
- Dynamic Prompt Assignment: To eliminate stragglers, the framework predicts upcoming response lengths using an Exponentially Weighted Moving Average (EWMA) of historical data. It then groups prompts of similar expected lengths into the same decoding actors, allowing short-workload actors to terminate and release their serverless resources early.
- Cut-and-Migrate Backoff: To handle stochastic variations where a prompt's response length is underestimated, RLHFless implements a runtime correction. If an actor is delayed by a long-running response, it cuts the generation at a predefined threshold and migrates the state to an available slot on another active actor.
- Cost-Aware Scaling & Locality Placement: Before each step, a planning module solves an optimization problem to find the "sweet spot" number of actors that minimizes both execution time and financial cost. It co-locates the prefill actor with the learner and prioritizes placing heavy decoding workloads on physically closer GPU nodes to hide network transfer latencies.
Datasets
The authors evaluated RLHFless using standard, widely accepted benchmarks that exhibit diverse response length characteristics:
- GSM8K: A dataset of high-school-level mathematical word problems, which typically trigger long, step-by-step reasoning chains.
- GPQA: A challenging, graduate-level Google-proof science question dataset.
- LiveCodeBench: A holistic benchmark for evaluating LLMs on complex coding tasks.
Large-scale scalability simulations were also conducted using synthetic workloads modeled after Llama2-70B training traces.
Key Insights and Ideas
- The Predictability of RLHF Workloads: Unlike online LLM serving where user queries are entirely unpredictable, RLHF training involves repeated sampling over a fixed dataset across multiple epochs. The authors realized that response lengths for specific prompts follow stable trends over training steps. This allows simple, model-free historical tracking (like EWMA) to act as an accurate workload predictor, avoiding the need for expensive auxiliary machine learning models.
- Serverless is a Natural Fit for Staged Pipelines: Because synchronous RLHF requires strict dependencies between sampling and learning, the "pay-as-you-go" granularity of serverless functions perfectly monetizes the idle time that plagues serverful setups. Spinning down generation actors the moment they finish directly cuts costs.
- Exploiting Length Diversity: Instead of treating variable response lengths as an scheduling nuisance, RLHFless turns it into an advantage. By sorting and grouping prompts by length, the system can systematically release blocks of GPUs early.
Key Results
- Overall Efficiency: Compared to the state-of-the-art serverful baseline (VERL), RLHFless achieves up to a 1.35× speedup and up to a 44.8% cost reduction on physical testbeds (AWS instances with NVIDIA L40S GPUs).
- Algorithm-Specific Gains: The framework delivers higher efficiency gains on Group Relative Policy Optimization (GRPO) than on Proximal Policy Optimization (PPO). This is because GRPO is highly sampling-heavy (generating many responses per prompt), leaving more room for KV cache deduplication and elastic scaling to eliminate waste.
- Scalability: In large-scale simulated environments scaling up to 20 nodes (160 NVIDIA H100 GPUs) running Llama2-70B, RLHFless maintained a stable scheduling overhead of under 30 ms while providing a 1.23× speedup and 38.7% cost reduction.
Limitations and Future Work
- Synchronous Focus: RLHFless is explicitly optimized for synchronous RLHF training. While its prompt assignment and actor-scaling strategies are conceptually orthogonal to asynchronous or off-policy RLHF frameworks, adapting them to handle stale data and continuous streaming updates remains future work.
- Hyperparameter Sensitivity: The framework relies on a few key hyperparameters, such as the cut-and-migrate threshold ($\tau$) and the actor-scaling weight ($\lambda$). While the paper provides robust default values ($\tau=0.7, \lambda=0.7$), these parameters may require initial profiling to find optimal trade-offs on highly customized hardware topologies.
Summary updated