Abstract
Normalizing Flows (NF) provide a robust framework for exact density estimation and efficient sampling. However, the manual design of NF architectures involves a difficult trade-off between expressivity and computational efficiency. Traditional Neural Architecture Search (NAS) methods are often inapplicable to NFs due to strict invertibility requirements and the performance gap introduced by binarizing continuous search weights. We propose AutoNF, the first differentiable NAS framework tailored specifically for NFs. By employing a novel mixture distribution formulation, AutoNF ensures mathematical invertibility throughout the search process. Furthermore, we provide a theoretical guarantee that our continuous relaxation naturally converges to the optimal discrete architecture, eliminating the binarization gap. Experimental results on standard benchmarks demonstrate that AutoNF discovers architectures that are significantly more efficient than hand-crafted baselines, achieving up to a 4.29x cost reduction with minimal performance loss.
Introduction
Normalizing Flows transform simple base distributions into complex target densities via a sequence of invertible mappings. Designing these sequences is a "Goldilocks" problem: overly complex flows are computationally prohibitive, while simpler ones may fail to capture data intricacies. While NAS has revolutionized architecture design for standard neural networks, its application to NFs is hindered by two primary challenges. First, the weighted sums of operations used in methods like DARTS do not preserve invertibility. Second, the heuristic binarization of "soft" weights typically leads to sub-optimal discrete models.
Methodology
AutoNF addresses these challenges through a paradigm shift in how architecture search is formulated for NFs.
Mixture Distribution Formulation
Instead of summing transformation functions directly—which would break invertibility—AutoNF operates in the distribution space. It defines the search as a mixture of probability density functions produced by candidate transformations. Since any mixture of valid probability distributions is itself a valid distribution, the model remains mathematically sound and implicitly invertible during the entire gradient-based search.
Optimal Continuous Relaxation
A key innovation of AutoNF is the mathematical proof that its specific optimization formulation (OP2) is equivalent to the original NP-hard combinatorial problem. As the search converges, the architecture weights naturally push toward binary values (0 or 1). This ensures that the final discrete model is the global optimum within the defined search space, bypassing the performance degradation common in traditional NAS binarization.
Decomposed Search and Cost Optimization
To manage the computational complexity of deep models, AutoNF utilizes a block-by-block cascading search. Additionally, it incorporates a user-defined cost factor ($\lambda$) in the objective function, allowing for a principled trade-off between density estimation performance (Negative Log-Likelihood) and computational cost.
Experimental Results
We evaluated AutoNF on standard density estimation benchmarks, including UCI datasets (POWER, GAS, HEPMASS, MINIBOONE) and BSDS300 image patches.
Performance-Cost Trade-offs
AutoNF consistently identifies architectures that define a superior Pareto front compared to hand-crafted baselines like MAF, RealNVP, and Rational Quadratic Spline flows (RQ-AF). For instance, on the POWER dataset, AutoNF discovered a model with a 4.29x reduction in computational cost compared to the best baseline while maintaining comparable NLL.
Automated Discovery
In several benchmarks, such as BSDS300 and MINIBOONE, AutoNF found architectures that outperformed standard human-designed models in both accuracy and efficiency. By adjusting $\lambda$, the framework can be tailored to specific hardware or latency constraints without manual intervention.
Conclusion
AutoNF represents a significant advancement in the automated design of Normalizing Flows. By reformulating NAS as a mixture modeling problem, it overcomes the fundamental constraints of invertibility and binarization sub-optimality. Future work will explore non-sequential supernet topologies and more advanced parameter-sharing schemes to further enhance search efficiency and model flexibility.
Summary updated