Deep learning traditionally relies on a simple, elegant mechanism: backpropagation. By calculating how a small change in a model's parameters affects its final output, optimization algorithms can adjust weights to minimize a loss function. This requires every step of the pipeline to be mathematically differentiable.
However, when we attempt to align generative models—such as Large Language Models (LLMs) or diffusion models—with complex real-world objectives, we run into a fundamental barrier. Many of the most critical objectives are "black boxes" that provide a score but offer no mathematical gradient. For example, evaluating whether a generated molecule will bind to a target protein requires running a physics-based docking simulator (Chen et al.). Determining if an image is aesthetically pleasing often relies on human feedback or proprietary vision-language APIs (Yeh). Optimizing code or data compression might depend on discrete file sizes or logical rules (the Wasserstein Gradient Flow framework).
In all these cases, the reward function is non-differentiable. Without a gradient, standard backpropagation cannot tell the model how to alter its weights to improve. To overcome this limitation, researchers have developed three core paradigms that bypass the differentiability constraint.
Paradigm 1: Bypassing the Reward Gradient via Policy Optimization
If you cannot compute the gradient of the reward function, the classical solution is to change what you are differentiating. Instead of calculating how changes in the model parameters affect the reward directly, policy gradient methods calculate how changes in the parameters affect the probability distribution of the generated outputs.
Imagine an agent trying to hit a target while blindfolded. The agent cannot see the map (no gradient), but an observer yells out a score after each attempt (the black-box reward). If the agent tries several random variations, it can look at the attempts that received the highest scores and shift its internal strategy to make those specific actions more likely in the future.
This log-derivative trick is the foundation of Reinforcement Learning (RL). As highlighted in early natural language processing applications, RL enables the optimization of models using qualitative or non-differentiable metrics like human feedback (Wang et al.).
Standard Gradient Descent:
Parameters ──> Output ──> Differentiable Loss ──> Gradient Backprop ──> Parameters
Policy Gradient (RL):
Parameters ──> Distribution ──> Sample Output ──> Black-Box Reward (Scalar)
│
Gradient of Log-Prob ────┴───────────────────────────────> Parameters
For continuous generative models, Black et al. applied this intuition to diffusion models with Denoising Diffusion Policy Optimization (DDPO). By framing the iterative denoising process as a multi-step Markov Decision Process (MDP), the authors showed that policy gradients can directly optimize diffusion weights for non-differentiable properties like image compressibility or prompt alignment. For discrete domains like DNA sequence design or text generation, where sampling steps are inherently non-smooth, Zekri et al. introduced Score Entropy Policy Optimization (SEPO) to enable stable policy gradient updates without requiring internal gradient information from the reward mechanism.
Paradigm 2: Inference-Time Steering and Particle Filtering
Updating the massive weight matrices of a generative model via reinforcement learning can be computationally expensive and prone to "reward hacking"—a failure mode where the model finds degenerate shortcuts that exploit the reward function's blind spots. A non-intrusive alternative is to leave the model's weights completely untouched and instead steer the generation process at inference time.
The intuition here is to treat the pre-trained generative model as a base proposal mechanism. Instead of generating a single output, the system simulates multiple parallel candidate trajectories (particles). At each step of the generation process, the black-box reward function evaluates the candidates. Paths that move toward low-reward regions are pruned, while paths that show promise are replicated.
Step t: [Path A] (Low Reward) ──> Pruned
[Path B] (High Reward) ──> Replicated ──> [Path B1], [Path B2]
[Path C] (Mid Reward) ──> Continued ──> [Path C1]
Yeh demonstrated this concept with Demon, a training-free framework that uses a sampling-and-weighting strategy to steer diffusion models using non-differentiable vision-language model APIs. To prevent the parallel paths from collapsing into a single identical image, Kim et al. utilized a tempered Sequential Monte Carlo (SMC) approach, balancing reward optimization with sample diversity.
When the generation space involves rigid structural or logical constraints, such as protein design or text generation, sequential selection can be enhanced by looking ahead. Guo et al. showed that integrating tree search allows the system to explore multiple potential futures, selecting the highest-reward branches at each step without ever needing to backpropagate through the constraint.
Paradigm 3: Bridging the Void with Differentiable Surrogates
A third paradigm seeks to restore the efficiency of direct gradient guidance by constructing a differentiable bridge over the black-box reward function. If the true reward function is a rugged, step-like, or opaque simulator that cannot provide gradients, we can train a secondary neural network—a surrogate—to mimic it. Because this surrogate is a neural network, it is fully differentiable.
The surrogate acts like a smooth blanket thrown over a jagged, rocky hill. The generative model can then use standard gradient ascent to climb the smooth surface of the surrogate, indirectly reaching the peak of the true black-box reward.
True Reward Landscape (Rugged/Discrete): └───┘└───┘ (No usable gradients)
Learned Surrogate (Smooth/Continuous): /───────\ (Provides clear gradients)
This approach is highly effective for accelerating ultra-fast generative models. For instance, Jia et al. developed LaSRO, which trains a differentiable surrogate reward model in the latent space to provide stable gradient guidance for lightning-fast, two-step diffusion models.
Similarly, Jensen et al. introduced Value Matching, which learns a differentiable value function that averages over the noise of a stochastic generative process. By tracking the expected future reward rather than the instantaneous black-box score, the model can navigate complex molecular and physical properties using smooth, deterministic gradient trajectories.