Phase-aware Training Schedule Simplifies Learning in Flow-Based Generative Models

Santiago Aranguri, Francesco InsullaSubmission to ICLR 2025 · Rejected

Abstract

We analyze the training of a two-layer autoencoder used to parameterize a flow-based generative model for sampling from a high-dimensional Gaussian mixture. Building on the work of Cui et al. (2024), we find that the phase where the high-level features are learnt during training disappears as the dimension goes to infinity without an appropriate time schedule. We introduce a time dilation that solves this problem. This enables us to characterize the learnt velocity field, finding a first phase where the high-level feature (asymmetry between modes) is learnt and a second phase where the low-level feature (distribution of each mode) is learnt. We find that the autoencoder representing the velocity field learns to simplify by estimating only the parameters relevant to the feature for each phase. Turning to real data, we propose a method that, for a given feature, finds intervals of time where training improves accuracy the most on that feature, and we provide an experiment on MNIST validating this approach.
BibTeX
@inproceedings{aranguri2025phase,
  author = {Santiago Aranguri and Francesco Insulla},
  title = {{Phase-aware Training Schedule Simplifies Learning in Flow-Based Generative Models}},
  booktitle = {FPI @ ICLR},
  year = {2025},
  url = {https://openreview.net/forum?id=ieRx3OFNIl},
}
Figure