ROCM: RLHF on consistency models

Shivanshu Shekhar, Tong ZhangarXiv 2025 · DOI · arXiv

Introduction

Diffusion models have fundamentally changed the landscape of generative AI, enabling high-fidelity image synthesis from text prompts. However, two significant hurdles remain: slow sampling speeds and the difficulty of fine-tuning these models using Reinforcement Learning from Human Feedback (RLHF). Standard RLHF techniques like Proximal Policy Optimization (PPO) are often computationally prohibitive for diffusion processes due to the need to backpropagate through numerous iterative steps or the reliance on high-variance gradient estimates.

To address these challenges, the authors introduce ROCM (RLHF on Consistency Models). This framework shifts the focus to Consistency Models (CMs), which are designed for rapid, few-step generation. By leveraging the unique properties of CMs, ROCM enables direct reward optimization, resulting in superior alignment with human preferences and significantly improved training efficiency.

Methodology: Direct Reward Optimization

The core innovation of ROCM lies in its approach to the RLHF objective. Unlike traditional diffusion models that require complex trajectories, Consistency Models generate samples in just 1 to 8 steps. This brevity allows for the use of the reparameterization trick.

Reparameterization and First-Order Gradients

By treating the generation process as a differentiable function, ROCM can backpropagate gradients directly from a reward model through the entire sampling chain. This transforms the reinforcement learning problem into a standard first-order optimization task, which is inherently more stable and efficient than policy-gradient methods like PPO.

Distributional Regularization via $f$-divergence

A common failure mode in RLHF is "reward hacking," where a model learns to exploit the reward function to achieve high scores while producing low-quality or nonsensical outputs. To mitigate this, ROCM incorporates $f$-divergence regularization. This term penalizes the model if its output distribution drifts too far from the original pre-trained distribution. The researchers evaluated several divergences—including KL-Divergence, Jensen-Shannon (JS), Hellinger distance, and Fisher divergence—finding that this regularization is crucial for maintaining image integrity and coherence.

Efficient Fine-tuning with LoRA

To keep the computational footprint low, ROCM utilizes Low-Rank Adaptation (LoRA). Instead of updating the entire model, only a small set of auxiliary parameters are trained, allowing the model to adapt to human preferences in as little as 15 GPU hours.

Experimental Setup and Results

The framework was evaluated using the Pick-a-Pic V1 dataset, utilizing 4,000 prompts for training and a dedicated validation set. Performance was measured against several established reward models: PickScore, HPSv2, CLIPScore, and Aesthetic Score.

Quantitative Performance

ROCM consistently outperformed existing baselines, including RLCM (PPO applied to consistency models) and DDPO (PPO for diffusion models). For example, when optimized for HPSv2, ROCM achieved an ImageReward score of 0.651, a substantial improvement over RLCM's 0.505.

Quantitative comparison of ROCM vs. RLCM across multiple reward metrics.

The win-rate analysis shown above highlights that ROCM-trained models are frequently preferred over RLCM across various dimensions, including prompt alignment and general visual appeal.

Qualitative Comparison

Visually, ROCM produces images that better adhere to complex prompts while maintaining high aesthetic quality. Compared to other alignment methods like DPOK or D3PO, ROCM demonstrates a better balance between following the reward signal and preserving the structural logic of the generated scenes.

Visual comparison of different alignment methods across various prompts. ROCM (top row) shows strong adherence to prompts and high visual quality.

Key Insights

  • Simplicity over Complexity: The research demonstrates that the heavy machinery of PPO is often unnecessary for consistency models. Direct backpropagation is simpler, faster, and less sensitive to hyperparameters.
  • Divergence Matters: The choice of regularization (e.g., Hellinger vs. KL) subtly influences the "style" of the model, affecting how it handles backgrounds and fine details.
  • Efficiency: The ability to achieve state-of-the-art alignment in 15 GPU hours makes RLHF accessible for a wider range of applications and researchers.

Limitations and Future Outlook

The primary constraint of ROCM is its requirement for differentiable reward signals. Because the method relies on first-order gradients, the reward function must be a differentiable neural network. Consequently, ROCM cannot currently optimize for non-differentiable objectives like image file size or discrete logical constraints.

Future research may focus on hybrid approaches that combine the efficiency of ROCM with the flexibility of policy-gradient methods for non-differentiable rewards, or extending the framework to video and 3D generation tasks.

Summary updated

BibTeX
@misc{shekhar2025rocm,
  author = {Shivanshu Shekhar and Tong Zhang},
  title = {{ROCM: RLHF on consistency models}},
  year = {2025},
  eprint = {2503.06171},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.06171},
}
Figure