MetroRLHF: Enabling Memory-Effective Training for On-Policy RLHF via Adaptive Sequence Streaming

Wei CuiER @ NeurIPS 2025 · OpenReview

TL;DR

This paper introduces MetroRLHF, a novel approach to make Reinforcement Learning from Human Feedback (RLHF) training for Large Language Models (LLMs) significantly more memory-efficient. By cleverly reusing computations from the inference phase and processing sequences in smaller, adaptive chunks, MetroRLHF drastically reduces peak memory usage without sacrificing training quality or introducing recomputation.

What Problem Does This Paper Address?

Training LLMs with RLHF is crucial for aligning them with human preferences, making them helpful and harmless. However, a major bottleneck is the enormous memory required during the policy model update phase of RLHF. This problem is exacerbated when dealing with long-form generation tasks, where sequences can become tens or even hundreds of thousands of tokens long. Current methods often resort to trade-offs like truncating sequences or using gradient checkpointing, which can degrade performance and context availability. MetroRLHF aims to solve this by enabling memory-efficient training without these compromises.

Methods

MetroRLHF tackles the memory issue by exploiting the Key-Value (K, V) cache generated during the RLHF inference phase. Normally, during training, the model needs to maintain dependencies across the entire sequence to backpropagate gradients. MetroRLHF's key innovation is to use the pre-computed K, V cache from inference to "decouple" these inter-token dependencies. This allows the training to process the sequence in smaller, adaptive subsequences.

The core of the method involves:

  • Adaptive Sequence Streaming: Instead of processing the entire long sequence at once, MetroRLHF breaks it down into smaller blocks (of size T). These blocks are processed in a reversed order (from "slow" to "high" indices).
  • Reusing Inference K, V Cache: The K, V cache, which stores intermediate attention computations from the inference phase, is reused. This eliminates the need to recompute these values during training, saving significant memory.
  • "Zero-Rewarding Shortcut": This optimization leverages the fact that in RLHF, only "answer" tokens contribute to the reward loss, not "question" tokens. MetroRLHF intelligently skips computations for blocks that consist entirely of question tokens, further reducing unnecessary work and memory usage.
  • Stream-able Attention Layer: The paper proposes a new attention layer architecture with "Instant FW" (forward pass) and "Stream-able BW" (backward pass) components. The forward pass computes local attention using the K, V cache, and the backward pass propagates local gradients while accumulating cross-block gradients.

Datasets

The paper evaluates MetroRLHF on the Qwen-3 models. Specific datasets for training or fine-tuning are not explicitly detailed beyond mentioning the use of these models.

Key Insights and Ideas

  • Inference-Phase Computations are Gold: The K, V cache, typically discarded after inference, holds valuable information that can be repurposed to drastically reduce training memory.
  • Breaking the Causal Chain: By using the K, V cache, the strict causal dependencies required for training can be relaxed, allowing for more flexible processing.
  • Targeted Optimization: Focusing on the specific loss structure of RLHF (i.e., only answer tokens contributing to loss) allows for intelligent skipping of computations, leading to efficiency gains.
  • Memory Reduction Without Recomputation: The method achieves significant memory savings while ensuring that the training process is semantically identical to full-sequence training, avoiding the pitfalls of methods that introduce recomputation.

Key Results

Experiments on Qwen-3 models demonstrate substantial memory reductions. MetroRLHF achieved peak training memory usage reductions ranging from 1/3.8 to 1/5.9 compared to full-sequence training. This enabled memory-effective and semantically reliable fine-tuning for LLMs, even on long-form generation tasks. The paper also notes that the "zero-rewarding shortcut" provides an average computational saving.

Limitations and Future Work

The authors acknowledge several limitations:

  • Zero Recomputation for On-Policy RLHF: MetroRLHF's efficiency relies on the pre-existing K, V cache from RLHF inference. This approach is not directly applicable to pre-training or supervised fine-tuning (SFT) where such a cache is unavailable, and recomputation would be necessary.
  • Requires Auto-Regressive LLMs: The method is designed for LLMs with causal (unidirectional) attention. Architectures with non-causal attention might not benefit as much, as their inter-token dependencies are not fully eliminated by inference.
  • Cuts Down Activation Memory Only: MetroRLHF primarily targets reducing the memory used by intermediate activations during training. It does not reduce the memory footprint of model parameters or optimizers.

Future work could explore extending this approach to pre-training and SFT, investigating its applicability to non-causal architectures, and combining it with other memory-saving techniques like mixed precision or QLoRA for even greater efficiency.

Summary updated

Abstract

Reinforcement learning from human feedback (RLHF) has become the standard post‑training technique for endowing large language models (LLMs) with helpful, harmless, and intent‑consistent behavior. In practice, however, its adoption is hampered by prohibitive memory consumption during the phase of the policy‑model update, especially when training on long‑form generation tasks. In this paper, we propose MetroRLHF, a memory‑efficient, on‑policy RLHF approach that exploits the inference-time computations to reduce the training-time memory budget and to skip unnecessary work. By re‑using the inference-phase materialized $K, V$ context, the inter‑token dependencies are freely removed that normally force the entire sequence to train in parallel. Building upon fine‑grained subsequence streaming, RLHF can train the productive tokens in an effective manner. This yields a training pipeline that matches the exact behavior of conventional full‑sequence RLHF while using less memory and incurring no arithmetic recomputation. Experiments on the Qwen‑3 models demonstrate that MetroRLHF rescheduled algorithm reduces peak training memory usage to 1/3.8 to 1/5.9, enabling not only memory-effective but also semantic-reliable fine‑tuning for LLM.
BibTeX
@inproceedings{cui2025metrorlhf,
  author = {Wei Cui},
  title = {{MetroRLHF: Enabling Memory-Effective Training for On-Policy RLHF via Adaptive Sequence Streaming}},
  booktitle = {ER @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=grcKmsx2it},
}
Figure