Abstract
Generative models based on Ordinary Differential Equations (ODEs) offer high-quality reconstruction but suffer from high computational costs during gradient-based optimization. We present FlowGrad, a framework that enables efficient, precise control over pre-trained ODE models. By utilizing a non-uniform discretization strategy that skips redundant computations along straight trajectory segments, FlowGrad reduces back-propagation time by approximately 90% while maintaining superior image manipulation quality compared to existing diffusion and GAN-based methods.
Introduction
While generative models like GANs and Diffusion models (SDEs) have reached incredible levels of realism, "steering" them to produce specific results remains a challenge. GANs are fast but notoriously difficult to invert, making it hard to find the exact latent code that reconstructs a specific real-world image. Diffusion models offer better reconstruction but are stochastic, making precise gradient-based control difficult. Often, controlling diffusion requires training specialized "noised" guidance models or fine-tuning the entire network for every new task.
ODE-based models, such as Rectified Flow, offer a middle ground: they are deterministic and allow for nearly perfect reconstruction. However, back-propagating gradients through an ODE trajectory involves "unrolling" a chain of many nested neural network passes. This is computationally expensive and quickly exhausts GPU memory, making standard optimization techniques impractical for long trajectories.
Methodology
FlowGrad treats controllable generation as an optimal control problem. The approach rests on two technical pillars:
1. Vector-Jacobian Products (VJP)
Instead of trying to compute a massive, memory-intensive Jacobian matrix for the entire ODE path, FlowGrad decomposes the back-propagation into a sequence of VJPs. This allows the gradient to flow from the output back to any intermediate time step efficiently using standard auto-differentiation frameworks like PyTorch.
2. Non-uniform Discretization
This is the core innovation for speed. The authors observed that many ODE trajectories are "straight" for large portions of their path. FlowGrad measures this "straightness" during the forward pass and groups these straight segments into a single discretization step during the backward pass. By only performing heavy gradient calculations where the trajectory "curves," the number of steps is drastically reduced.
Key Insights and Results
- The "Straightness" Advantage: The most significant finding is that uniform steps are unnecessary for calculating gradients. If the model's velocity is relatively constant, a single large step is a near-perfect approximation of many tiny steps. This allows FlowGrad to ignore approximately 90% of the trajectory during back-propagation with negligible loss in accuracy.
- Efficiency: On a TITAN XP GPU, standard back-propagation through a 100-step trajectory took 17.7 seconds. FlowGrad’s accelerated version reduced this to just 1.1 to 1.4 seconds.
- Quality: In text-guided image manipulation (e.g., changing facial expressions), FlowGrad outperformed state-of-the-art baselines like DiffusionCLIP and StyleCLIP. It achieved higher identity similarity and better LPIPS scores (preserving image structure) while maintaining high alignment with text prompts.
- Memory Optimization: FlowGrad allows for optimization on 12GB GPUs that would typically crash when attempting direct back-propagation through more than 4 steps of a deep ODE model.
Conclusion
FlowGrad demonstrates that efficient gradient-based control of ODE models is possible by exploiting the geometric properties of their trajectories. While currently focused on Euler solvers and image manipulation, future work may extend this to higher-order solvers (like Runge-Kutta) and other modalities such as video generation or molecular modeling. The framework provides a powerful tool for precise, deterministic editing of high-resolution generative outputs.
Summary updated