Abstract
Generative models like diffusion and flow matching have shown immense promise in modeling complex data distributions. However, guiding these models toward specific high-value outcomes—such as high-reward actions in robotics or optimal molecular properties—traditionally requires complex auxiliary models. We present Energy-Weighted Flow Matching (EFM), a novel framework that eliminates the need for these "middleman" models by incorporating energy guidance directly into the training objective. Applied to offline reinforcement learning (RL), our algorithm, Q-weighted Iterative Policy Optimization (QIPO), achieves state-of-the-art performance with significant improvements in sampling speed and accuracy.
Introduction
Generative models are typically trained to replicate a training distribution. In many scientific and engineering domains, however, the goal is not just to replicate the data but to optimize for specific properties. This is known as energy-guided generation, where an "energy function" defines the desirability of a data point.
Existing methods for energy guidance often rely on auxiliary models—separate neural networks trained to provide gradients or classifications to steer the generative process. These auxiliary models introduce significant overhead, approximation errors, and computational bottlenecks during sampling. Energy-Weighted Flow Matching (EFM) addresses these issues by baking the guidance directly into the generative model's weights during training.
Methods: Guidance Without the Middleman
The fundamental innovation of EFM (and its diffusion counterpart, ED) is a re-weighting of the training loss. Instead of training on all data points equally, the loss function is weighted by the energy function.
The Weighted Objective
In EFM, data points that correspond to lower energy (higher desirability) are given higher importance in the loss function. The authors provide a mathematical proof that a model trained with this weighted objective will naturally converge to the exact energy-guided distribution. This removes the need for backpropagating through a separate value network or classifier during the generation phase.
QIPO for Offline Reinforcement Learning
In the context of Offline RL, the energy function is defined by the negative Q-value (the expected future reward). The resulting algorithm, Q-weighted Iterative Policy Optimization (QIPO), iteratively refines the policy. By using the weighted training approach, QIPO can "sharpen" the policy to focus on high-reward actions even when the original dataset contains suboptimal or noisy behavior.
Experimental Results
The researchers evaluated EFM and QIPO across several challenging benchmarks, including the D4RL navigation tasks, image synthesis, and molecular design.
Performance and Efficiency
In the D4RL AntMaze tasks, QIPO-OT (the Flow Matching variant) and QIPO-Diff consistently outperformed established baselines like QGPO and Diffusion-QL. Notably, because the guidance is integrated into the model weights, the sampling process is significantly more efficient. QIPO-OT demonstrated a 63% speedup in action generation compared to previous state-of-the-art guided diffusion methods, making it highly suitable for real-time robotics applications.
Robustness Across Domains
Beyond RL, EFM successfully shifted distributions in image generation (e.g., generating butterflies of specific colors) and molecular design (e.g., optimizing for isotropic polarizability). In these tasks, EFM proved to be more effective at targeting the desired distribution than heuristic methods like Classifier-Free Guidance.
Conclusion and Future Work
Energy-Weighted Flow Matching represents a significant step forward in making guided generative models more practical and theoretically sound. By eliminating auxiliary models, EFM simplifies the training pipeline and drastically reduces inference time. While currently focused on offline settings, future research will explore extending this framework to online reinforcement learning, where models must learn and adapt through continuous interaction with their environment.
Summary updated