Knowledge Transfer for Conditional Flow Matching: Frozen Estimation and Adaptation

Yian Huang, Jian HuangarXiv 2026 · arXiv

TL;DR

This paper introduces a robust transfer learning framework for Conditional Flow Matching (CFM), allowing researchers to build high-quality generative models even with sparse target data. By either "freezing" a source-learned representation or "adapting" it with a functional penalty, the authors provide a mathematically grounded way to leverage related populations while including a "fail-safe" validation step to prevent negative transfer.

What Problem Does This Paper Address?

Conditional generative modeling—learning to produce a response (like an image or a physical property) given certain covariates (like a text prompt or chemical composition)—is data-hungry. In many specialized fields, we have a "source" dataset with plenty of samples but a "target" task with very few.

The challenge is that the source and target might have different response distributions or even different dimensions. Standard transfer learning often fails here because it assumes too much similarity. Furthermore, "freezing" a pre-trained model might leave out crucial target-specific information, while "fine-tuning" everything from scratch on a small dataset leads to overfitting. The authors aim to find the "Goldilocks zone" of representation transfer for Flow Matching, a modern alternative to Diffusion models that learns to transport a simple base distribution to a complex target via Ordinary Differential Equations (ODEs).

Methods

The authors propose two primary procedures under the Transfer Flow Matching (TFM) umbrella:

  1. TFM-Freeze: You take an encoder (representation) learned from the source data and keep it fixed. You only train a new "velocity head" on the target data. This is computationally cheap and reduces the number of parameters to learn.
  2. TFM-Adapt: You jointly learn the velocity head and an updated representation. To prevent the model from forgetting the useful source knowledge, they use a functional anchoring penalty. Unlike standard weight regularization, this penalty ensures the output of the new encoder stays close to the source encoder's output on a set of anchor points.

Crucially, they add a Distributional Validation step. Using an independent validation set and an "energy-score" rule, the system decides whether to use the transferred model or a "target-only" model trained from scratch. This acts as a safeguard against negative transfer (where using source data actually makes the model worse).

Datasets

The paper evaluates these methods on a mix of synthetic and real-world data:

  • Simulated Environments: Designed with nonlinear representations, heteroscedasticity (varying noise levels), and mixture responses to test the limits of the risk bounds.
  • Superconducting Materials (UCI): Predicting the critical temperature of materials. The source domain consists of copper-oxide compounds without iron, while the target domain contains iron-bearing compounds without copper.
  • Diamond Prices (ggplot2): Predicting log-prices based on physical dimensions and quality. The source domain uses "Premium" and "Very Good" cuts, while the target focuses on "Ideal" cuts.

Key Insights and Ideas

  • Smooth Correctability: The authors prove that if the difference between the source and target representations can be described by a "smooth correction" in a few dimensions, adaptation is significantly more efficient than learning a new representation from scratch.
  • The Cost of Freezing: They derive an information lower bound showing that any "frozen" model has a fundamental limit on its accuracy if the source representation doesn't capture specific target-mean information. Adaptation can break through this floor.
  • Oracle Inequalities: The proposed validation rule isn't just a heuristic; the authors provide an "oracle inequality," proving that the selection procedure will perform nearly as well as the best possible candidate in the dictionary, with high probability.
  • Flow Stability: A significant portion of the theory accounts for the "numerical integration" error. Since Flow Matching relies on ODE solvers (like Euler steps), the risk bounds explicitly include the number of steps taken during generation.

Key Results

  • Simulation Gains: In related tasks, TFM-Adapt reduced energy discrepancy by 26.4% compared to the best non-adaptive method for 1D responses and 18.2% for 5D responses.
  • Real-World Success: On the Diamonds dataset, TFM-Adapt improved on target-only flow matching by 8.05%.
  • Safety Mechanism: In the Materials application—where the source and target were intentionally different enough to risk negative transfer—the validation safeguard correctly rejected the transfer in 100% of repetitions, reverting to the target-only model and avoiding a performance hit.

Limitations and Future Work

  • ReLU Specificity: The theoretical bounds are derived specifically for ReLU neural network classes. While ReLU is standard, extending these guarantees to other architectures (like Transformers or Mamba blocks) remains an open question.
  • High-Dimensional Context: While the theory handles high-dimensional covariates, the "curse of dimensionality" still affects the rates of convergence. The benefits of transfer are most pronounced when the underlying "feature map" is lower-dimensional.
  • Computational Overhead: Joint adaptation (TFM-Adapt) is more computationally intensive than freezing. Future work could explore more efficient ways to implement the functional anchoring penalty in large-scale models.

For those interested in the deep math, the full paper provides end-to-end neural risk bounds in the Appendices, accounting for everything from source estimation error to the discretization of the ODE solver.

Summary updated

Abstract

Conditional generative modeling with limited target data can benefit from representations learned in a related source population. We develop a transfer learning framework for conditional flow matching that accommodates distinct source and target response distributions and dimensions. Two procedures exploit a representation learned from source data: TFM-Freeze estimates a target velocity head with the representation fixed, whereas TFM-Adapt jointly learns the head and a representation adaptation under a functional anchoring penalty. For explicit neural network classes, we establish bounds on conditional squared Wasserstein risk that account for source estimation, target fitting, representation mismatch, and numerical integration. When the source representation admits a smooth correction, adaptation replaces the frozen mismatch contribution by the cost of learning the correction and a controllable regularization bias. An information lower bound identifies regimes in which adaptation attains a strictly faster conditional squared Wasserstein risk rate than any generator restricted to the frozen context information. We also develop independent validation procedure for selecting among fitted conditional generators. We show that the selection procedure satisfies an oracle inequality, and the switching rule guarantees, with high probability, no increase in conditional energy risk relative to a target-only generator. Simulations and real data applications illustrate the gains from our proposed methods and the effectiveness of target validation in guarding against negative transfer.
BibTeX
@misc{huang2026knowledge,
  author = {Yian Huang and Jian Huang},
  title = {{Knowledge Transfer for Conditional Flow Matching: Frozen Estimation and Adaptation}},
  year = {2026},
  eprint = {2611.15150},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15150},
}
Figure