Tilt Matching for Scalable Sampling and Fine-Tuning

Peter Potaptchik, Cheuk Kit Lee, Michael S. Albergo

Abstract

Generative modeling has seen significant success when large datasets are available. However, steering these models using unnormalized densities (energy functions) or scalar rewards (human preferences) remains a significant challenge. Traditional approaches often rely on expensive trajectory-based backpropagation or require differentiable reward functions. This paper introduces Tilt Matching (TM), a scalable framework that leverages a novel "Covariance ODE" to evolve model velocity fields under a reward "tilt." By reducing the problem to a regression task on scalar rewards, Tilt Matching avoids the computational pitfalls of previous methods and achieves state-of-the-art performance in both statistical physics and image generation tasks.

1. Introduction

Generative frameworks like Diffusion and Flow Matching have revolutionized the synthesis of complex data. Yet, many critical applications—ranging from molecular simulation to human-aligned AI—require models to be steered by scalar feedback rather than static datasets. This process, known as tilting, involves adapting a pre-trained model to a target distribution defined by a reward or energy function.

Existing tilting methods generally suffer from two major bottlenecks. Trajectory-based methods require backpropagating through ODE/SDE solvers, which is memory-intensive and numerically unstable. Alternatively, gradient-based methods require the reward function to be differentiable, which is often impossible when dealing with black-box simulators or complex human-preference models. Tilt Matching addresses these issues by providing a "data-less" steering mechanism that is as simple and scalable as standard generative training.

2. Methodology

2.1 The Covariance ODE

Tilt Matching operates within the framework of stochastic interpolants. The core innovation is the derivation of the Covariance ODE, which describes how the model's velocity field ($b$) evolves as the reward "tilt" (parameterized by $a$) is annealed from 0 to 1.

The authors prove that the infinitesimal change in the velocity field is equivalent to the conditional covariance between the model's internal dynamics and the reward signal. This discovery allows the model to update its velocity field by observing how specific directions in latent space correlate with higher rewards, effectively pushing the dynamics toward high-reward regions without requiring reward gradients.

2.2 ETM and ITM

The paper proposes two implementations of this framework:

  • Explicit Tilt Matching (ETM): This uses a first-order Euler discretization of the Covariance ODE. It is straightforward to implement but subject to discretization errors over the annealing path.
  • Implicit Tilt Matching (ITM): A more robust fixed-point iteration method that captures all higher-order cumulants of the reward. ITM is particularly powerful as it eliminates the discretization error associated with annealing steps and exhibits strictly lower variance than standard flow matching by centering updates on conditional means.

Crucially, both methods rely solely on regression against scalar rewards, bypassing the need for trajectory simulation or differentiable rewards during the inner training loop.

3. Experimental Evaluation

3.1 Statistical Physics: Lennard-Jones Potentials

To test the ability of Tilt Matching to capture complex, multi-modal energy landscapes, the researchers evaluated the model on Lennard-Jones (LJ) particle systems (LJ-13 and LJ-55). These high-dimensional benchmarks are notoriously difficult for standard samplers.

Tilt Matching achieved an Effective Sample Size (ESS) of 0.507 for LJ-13, more than doubling the performance of the next best method (0.231). Furthermore, the method successfully scaled to LJ-55, a 165-dimensional system with complex symmetries where many trajectory-based methods fail to converge.

3.2 AI Content Generation: Fine-Tuning Stable Diffusion

The framework was also applied to fine-tuning Stable Diffusion 1.5 using the ImageReward dataset. The goal was to align the model with human aesthetic preferences and prompt adherence. Tilt Matching outperformed "Adjoint Matching" across several key metrics, including ImageReward scores (0.4465 vs 0.2170) and CLIPScore, while maintaining superior sample diversity.

4. Discussion and Key Insights

  • Covariance as a Natural Guide: The link between velocity updates and conditional covariance provides a clean, intuitive mechanism for steering. It identifies latent directions that correlate with success and amplifies them.
  • Optimal Control Bridge: Tilt Matching provides a formal link to stochastic optimal control, specifically recovering Doob’s h-transform. It achieves optimal transport results using a much simpler regression-based architecture.
  • Stability: Unlike competing methods that require sensitive reward-scaling hyperparameters (e.g., $\lambda$), Tilt Matching works effectively with the raw reward signal, suggesting a more stable integration of feedback.

5. Limitations and Future Work

While Tilt Matching represents a significant step forward, several areas remain for exploration. The user must still define an annealing schedule for the parameter $a$, and finding the optimal path for this schedule is an open research question. Additionally, like all reward-based fine-tuning, there is a risk of "reward hacking" or mode collapse if the model over-optimizes for specific peaks in the reward landscape. Future work may extend Tilt Matching to "few-step flow maps" like Consistency Models to enable high-speed, steered generation.

6. Conclusion

Tilt Matching offers a scalable, gradient-free, and trajectory-free solution for steering generative models. By transforming the tilting problem into a series of regression tasks guided by the Covariance ODE, it provides a robust framework for both scientific sampling and the alignment of large-scale generative AI.

Summary updated

BibTeX
@inproceedings{potaptchik2026tilt,
  author = {Peter Potaptchik and Cheuk Kit Lee and Michael S. Albergo},
  title = {{Tilt Matching for Scalable Sampling and Fine-Tuning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=dQA4Gjt4KU},
}
Figure