Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control

Huayu Chen, Kaiwen Zheng, Hang Su, Jun Zhu

Introduction

In the landscape of Offline Reinforcement Learning (RL), diffusion models have emerged as a powerful tool for capturing the complex, multi-modal distributions of expert behavior. However, a significant gap remains: while these models excel at imitation, refining them into optimized policies typically requires computationally intensive "guidance" or training from scratch. This paper introduces Efficient Diffusion Alignment (EDA), a framework that adapts the "pretrain-then-align" paradigm—highly successful in Large Language Models (LLMs)—to the domain of continuous control.

The Challenge: The Density Gap

Offline RL involves learning optimal policies from static, often suboptimal, datasets. Diffusion models are the current gold standard for this task because they can model the diverse ways humans or robots might approach a problem. However, turning that imitation into an optimized policy is difficult because standard diffusion models lack a "tractable density." Mathematically, it is hard to calculate the exact probability of an action, which is a prerequisite for standard alignment algorithms like RLHF or Direct Preference Optimization (DPO). Consequently, researchers often have to choose between the flexibility of diffusion and the optimizability of simpler models.

The EDA Framework

EDA bridges this gap through a two-stage "pretrain-then-align" framework designed for efficiency and performance.

1. Behavior Pretraining via Bottleneck Diffusion

To solve the tractable density problem, the authors redefine the architecture of the diffusion model. Instead of using a standard network that predicts noise, they represent the policy as the derivative of a scalar neural network (a "bottleneck") with respect to the action. This scalar network effectively learns the unnormalized density (energy) of the behavior. By constraining the model in this way, it becomes an Energy-Based Model (EBM), which provides the mathematical "log-probability" foundation needed for alignment.

2. Policy Fine-tuning via Q-function Alignment

Once a behavior model is pretrained, it must be aligned with a reward signal. Inspired by DPO, EDA uses a contrastive loss. The system samples $K$ candidate actions from the pretrained model and uses a learned Q-function to identify which actions are most valuable. The model is then fine-tuned to increase the log-probability ratio between the new policy and the old behavior for these high-value actions. This allows the model to "remember" the diverse behaviors it learned during pretraining while quickly shifting its preference toward high-reward outcomes.

Experimental Evaluation

The researchers evaluated EDA on the standard D4RL benchmark, covering a wide range of challenges:

  • MuJoCo Locomotion: Tasks such as HalfCheetah, Hopper, and Walker2d with varying data quality.
  • AntMaze: Navigation tasks requiring long-term planning and reasoning.
  • FrankaKitchen: Complex, multi-step robotic manipulation tasks.

Key Insights and Results

EDA demonstrates that the "pretrain-then-align" strategy is highly effective for continuous control. Key findings include:

  • State-of-the-Art Performance: EDA outperformed major baselines, including IDQL, Diffusion-QL, and QGPO, across the D4RL suite.
  • Extreme Data Efficiency: The model maintained 95% of its performance even when the fine-tuning stage was restricted to only 1% of the labeled data. This suggests that the model can adapt to new tasks with very few reward annotations.
  • Rapid Convergence: While typical offline RL models may require 1 million gradient steps, EDA converges in approximately 20,000 steps—a 50x speedup in the fine-tuning phase.

Limitations and Future Directions

Despite its success, EDA relies on a pretrained critic (Q-function), meaning the final policy's quality is capped by the accuracy of that critic. Additionally, the performance is sensitive to the temperature coefficient ($\beta$), which balances exploration of the original behavior versus optimization of the reward.

Looking forward, the authors view EDA as a stepping stone toward Foundation Models for Control. In this vision, a massive multi-task behavior model could be rapidly aligned to specific downstream tasks with minimal effort, mirroring the evolution of LLMs in natural language processing.

Summary updated

Abstract

Drawing upon recent advances in language model alignment, we formulate offline Reinforcement Learning as a two-stage optimization problem: First pretraining expressive generative policies on reward-free behavior datasets, then finetuning these policies to align with task-specific annotations like Q-values. This strategy allows us to leverage abundant and diverse behavior data to enhance generalization and enable rapid adaptation to downstream tasks using minimal annotations. In particular, we introduce Efficient Diffusion Alignment (EDA) for solving continuous control problems. EDA utilizes diffusion models for behavior modeling. However, unlike previous approaches, we represent diffusion policies as the derivative of a scalar neural network with respect to action inputs. This representation is critical because it enables direct density calculation for diffusion models, making them compatible with existing LLM alignment theories. During policy fine-tuning, we extend preference-based alignment methods like Direct Preference Optimization (DPO) to align diffusion behaviors with continuous Q-functions. Our evaluation on the D4RL benchmark shows that EDA exceeds all baseline methods in overall performance. Notably, EDA maintains about 95\% of performance and still outperforms several baselines given only 1\% of Q-labelled data during fine-tuning.
BibTeX
@inproceedings{chen2024aligning,
  author = {Huayu Chen and Kaiwen Zheng and Hang Su and Jun Zhu},
  title = {{Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Wd1DFLUp1M},
}
Figure