PaddingFlow: Improving Normalizing Flows with Padding-Dimensional Noise

Qinglong Meng, Chongkun Xia, Xueqian WangarXiv 2024 · DOI · arXiv

Abstract

Normalizing Flows (NFs) are powerful generative models that map simple distributions to complex data distributions via invertible transformations. However, they often struggle when data resides on a low-dimensional manifold or when dealing with discrete values, leading to training instability or "collapsed" distributions. This paper introduces PaddingFlow, a method that addresses these hurdles by concatenating extra dimensions of Gaussian noise to the input data. This "padding" enables the flow to model complex distributions more effectively without the high computational cost of variational methods or the bias introduced by standard uniform dequantization.

Introduction: The Challenge of Manifold Mismatch

Normalizing Flows face two major hurdles in practice. First is the Manifold Mismatch: real-world data often lives on a low-dimensional manifold embedded in a high-dimensional space (e.g., a 2D curve in 3D space). Since NFs typically require the latent and data dimensions to be equal, they struggle to "stretch" a high-dimensional volume onto a thin surface. Second is Discrete Data Collapse: when training on discrete data like pixel values, NFs can collapse into a "mixture of point masses," where the model assigns infinite density to specific points, ruining generalization.

Existing solutions like Variational Dequantization are computationally expensive, requiring an extra model to generate noise, while Uniform Dequantization can lead to biased estimations.

Methodology: Padding-Dimensional Noise

The core idea of PaddingFlow is to add noise next to the data rather than just to the data. This approach satisfies key criteria: ease of implementation, preservation of data distribution, unbiased estimation, and low computational cost.

Conceptual overview of the PaddingFlow framework, showing the addition of noise and padding dimensions to the data distribution.

The transformation process involves:

  1. Data Noise ($\epsilon_d$): A small amount of Gaussian noise is added to the original data dimensions to handle discreteness.
  2. Padding Noise ($\epsilon_p$): Extra dimensions of Gaussian noise are concatenated to the data vector.
  3. The Augmented Input: The input becomes $x' = (x + \epsilon_d, \epsilon_p)$. The Normalizing Flow is then trained on this augmented space.

To sample from the model, the flow is run in reverse, and the padding dimensions are simply discarded. Because the padding noise is independent and its distribution is known, it does not distort the marginal distribution of the actual data.

Experimental Evaluation

The authors validated PaddingFlow across three distinct domains: tabular datasets (UCI benchmarks), image datasets (MNIST, Omniglot), and robotics (Inverse Kinematics for a Panda manipulator).

2D Toy Tasks

On "circles" and "sines" distributions, PaddingFlow successfully captured the manifold structure. In contrast, standard models like FFJORD and specialized ones like SoftFlow showed significant artifacts or failures, particularly in conditional settings.

Visual comparison of PaddingFlow against FFJORD and SoftFlow on 2D toy datasets (circles and sines), demonstrating superior manifold capture.

Quantitative Results

  • Tabular Performance: PaddingFlow consistently outperformed FFJORD across five UCI datasets. On the GAS dataset, it reduced the Chamfer Distance (CD) from 1.29 to 0.890.
  • Image Quality: In VAE experiments, PaddingFlow achieved lower Cross Entropy and better image reconstruction compared to standard FFJORD-based VAEs.
  • Robotics (IK): In Inverse Kinematics tasks, PaddingFlow achieved a lower Position Error (5.96 mm) and Angular Error (0.621 deg) compared to GLOW and SoftFlow.

Discussion and Key Insights

PaddingFlow provides the "room" necessary for the flow to map a high-dimensional latent space to a lower-dimensional data manifold without losing invertibility or numerical stability. Mathematically, the authors prove that while standard uniform dequantization is biased, PaddingFlow provides an unbiased estimation of the data.

A significant advantage of PaddingFlow is its simplicity. Unlike SoftFlow or Variational Dequantization, which require complex conditional architectures or auxiliary networks, PaddingFlow only requires adjusting the input/output dimensions of the existing NF architecture.

Limitations

Selecting the optimal number of padding dimensions ($p$) and the variance of the noise can be challenging. The authors found it difficult to find a "one-size-fits-all" $p$ for every dataset, suggesting that future work could involve automated methods to determine optimal padding based on the intrinsic dimensionality of the data manifold.

Summary updated

BibTeX
@misc{meng2024paddingflow,
  author = {Qinglong Meng and Chongkun Xia and Xueqian Wang},
  title = {{PaddingFlow: Improving Normalizing Flows with Padding-Dimensional Noise}},
  year = {2024},
  eprint = {2403.08216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.08216},
}
Figure