AutoNF: Automated Architecture Optimization of Normalizing Flows with Unconstrained Continuous Relaxation Admitting Optimal Discrete Solution

Yu Wang, Ján Drgona, Jiaxin Zhang, Karthik Somayaji Nanjangud Suryanarayana, Malachi Schram, Frank Liu, Peng LiAAAI 2023 · DOI

Abstract

Normalizing Flows (NF) provide a robust framework for exact density estimation and efficient sampling. However, the manual design of NF architectures involves a difficult trade-off between expressivity and computational efficiency. Traditional Neural Architecture Search (NAS) methods are often inapplicable to NFs due to strict invertibility requirements and the performance gap introduced by binarizing continuous search weights. We propose AutoNF, the first differentiable NAS framework tailored specifically for NFs. By employing a novel mixture distribution formulation, AutoNF ensures mathematical invertibility throughout the search process. Furthermore, we provide a theoretical guarantee that our continuous relaxation naturally converges to the optimal discrete architecture, eliminating the binarization gap. Experimental results on standard benchmarks demonstrate that AutoNF discovers architectures that are significantly more efficient than hand-crafted baselines, achieving up to a 4.29x cost reduction with minimal performance loss.

Introduction

Normalizing Flows transform simple base distributions into complex target densities via a sequence of invertible mappings. Designing these sequences is a "Goldilocks" problem: overly complex flows are computationally prohibitive, while simpler ones may fail to capture data intricacies. While NAS has revolutionized architecture design for standard neural networks, its application to NFs is hindered by two primary challenges. First, the weighted sums of operations used in methods like DARTS do not preserve invertibility. Second, the heuristic binarization of "soft" weights typically leads to sub-optimal discrete models.

Methodology

AutoNF addresses these challenges through a paradigm shift in how architecture search is formulated for NFs.

Mixture Distribution Formulation

Instead of summing transformation functions directly—which would break invertibility—AutoNF operates in the distribution space. It defines the search as a mixture of probability density functions produced by candidate transformations. Since any mixture of valid probability distributions is itself a valid distribution, the model remains mathematically sound and implicitly invertible during the entire gradient-based search.

AutoNF architecture search using mixture distribution formulation and the resulting loss landscape showing the elimination of the binarization gap.

Optimal Continuous Relaxation

A key innovation of AutoNF is the mathematical proof that its specific optimization formulation (OP2) is equivalent to the original NP-hard combinatorial problem. As the search converges, the architecture weights naturally push toward binary values (0 or 1). This ensures that the final discrete model is the global optimum within the defined search space, bypassing the performance degradation common in traditional NAS binarization.

Decomposed Search and Cost Optimization

To manage the computational complexity of deep models, AutoNF utilizes a block-by-block cascading search. Additionally, it incorporates a user-defined cost factor ($\lambda$) in the objective function, allowing for a principled trade-off between density estimation performance (Negative Log-Likelihood) and computational cost.

Experimental Results

We evaluated AutoNF on standard density estimation benchmarks, including UCI datasets (POWER, GAS, HEPMASS, MINIBOONE) and BSDS300 image patches.

Performance-Cost Trade-offs

AutoNF consistently identifies architectures that define a superior Pareto front compared to hand-crafted baselines like MAF, RealNVP, and Rational Quadratic Spline flows (RQ-AF). For instance, on the POWER dataset, AutoNF discovered a model with a 4.29x reduction in computational cost compared to the best baseline while maintaining comparable NLL.

Performance-cost trade-offs in AutoNF across different lambda values and comparison with baselines.

Automated Discovery

In several benchmarks, such as BSDS300 and MINIBOONE, AutoNF found architectures that outperformed standard human-designed models in both accuracy and efficiency. By adjusting $\lambda$, the framework can be tailored to specific hardware or latency constraints without manual intervention.

Conclusion

AutoNF represents a significant advancement in the automated design of Normalizing Flows. By reformulating NAS as a mixture modeling problem, it overcomes the fundamental constraints of invertibility and binarization sub-optimality. Future work will explore non-sequential supernet topologies and more advanced parameter-sharing schemes to further enhance search efficiency and model flexibility.

Summary updated

BibTeX
@inproceedings{wang2023autonf,
  author = {Yu Wang and Ján Drgona and Jiaxin Zhang and Karthik Somayaji Nanjangud Suryanarayana and Malachi Schram and Frank Liu and Peng Li},
  title = {{AutoNF: Automated Architecture Optimization of Normalizing Flows with Unconstrained Continuous Relaxation Admitting Optimal Discrete Solution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i8.26220},
}
Figure