How to Train Your FALCON: Learning Log-Concave Densities with Energy-Based Neural Networks

Alexander Lin, Demba E. BaAABI 2023 · OpenReview

TL;DR

This paper introduces FALCON, a novel method for learning log-concave densities (LCDs) using energy-based neural networks. FALCON scales LCD estimation to high dimensions, achieving significant speedups and parameter reductions compared to existing methods, and demonstrates its effectiveness on a challenging real-world dataset.

What Problem Does This Paper Address?

Estimating log-concave densities (LCDs) is a fundamental problem in statistics. An LCD is a probability distribution whose logarithm is a concave function. This class of distributions is a natural, non-parametric generalization of Gaussians, offering greater flexibility in modeling data. However, finding the "best" LCD that fits observed data (the maximum likelihood estimator, or MLE) is computationally challenging. Existing algorithms for LCD estimation struggle with high-dimensional data (typically limited to fewer than 10 dimensions) due to their heavy computational demands. This limits their applicability to many real-world datasets where features can number in the tens or even thousands.

Methods

The core innovation of FALCON lies in reframing the non-parametric LCD estimation problem into a parametric one solvable with deep learning. Instead of directly searching over the space of all possible LCDs, FALCON uses an energy-based model (EBM). Specifically, it parameterizes the negative log-density of an LCD using an input-convex neural network (ICNN).

Here's a breakdown:

  • Energy-Based Models (EBMs): These models define a probability distribution by an "energy function" $E_\theta(x)$, where $\theta$ represents the model's parameters. The probability density is then proportional to $exp(-E_\theta(x))$.
  • Input-Convex Neural Networks (ICNNs): These are specialized neural networks designed to guarantee that their output is a convex function of their input. This property is crucial because for a distribution to be log-concave, its negative log-density must be convex. By using an ICNN to define the energy function $E_\theta(x)$, FALCON ensures the learned density is indeed log-concave.
  • Training: FALCON is trained using stochastic gradient descent (SGD) to maximize the log-likelihood of the observed data. This process effectively optimizes the weights of the ICNN, turning the non-parametric search into a parametric optimization problem.
  • Sampling: To draw samples from the learned FALCON distribution, the paper employs the Metropolis-Adjusted Langevin Algorithm (MALA), which is well-suited for sampling from log-concave distributions and scales efficiently with dimension.

Datasets

The paper uses two types of datasets:

  • Simulated Data: Synthetic datasets are generated from multivariate Gaussian distributions in dimensions ranging from 2 to 8. This allows for a controlled comparison against established LCD estimation algorithms.
  • Real Protein Expression Data: A real-world dataset from the UCI machine learning repository is used, featuring 77 dimensions (proteins) and 552 samples. This dataset is particularly challenging due to its high dimensionality, making it infeasible for previous LCD estimation methods.

Key Insights and Ideas

  • Bridging Statistics and Deep Learning: FALCON elegantly bridges the gap between statistical challenges in non-parametric density estimation (like LCDs) and the powerful learning capabilities of deep neural networks.
  • Scalability through Parameterization: The key insight is that by parameterizing the negative log-density with an ICNN, the computationally intractable non-parametric problem becomes a tractable parametric one, enabling scalability to high dimensions.
  • Efficiency Gains: Leveraging GPUs and deep learning infrastructure allows FALCON to be orders of magnitude faster and require significantly fewer parameters than traditional methods.
  • Expressive Power of EBMs: The work highlights how EBMs, when combined with specific architectures like ICNNs, can be powerful tools for learning complex, constrained probability distributions, not just for generation but for accurate density estimation.
  • Flexibility in Real-World Applications: The successful application to high-dimensional protein expression data demonstrates FALCON's practical utility beyond theoretical benchmarks, particularly in tasks like missing data imputation.

Key Results

  • Speed: FALCON is up to 1,900x faster than LogConcDEAD and 120x faster than fmlogcondens on simulated data.
  • Parameters: FALCON uses up to 290x fewer parameters than LogConcDEAD and up to 750x fewer parameters than fmlogcondens.
  • Accuracy: On simulated data, FALCON achieves negative log-likelihoods with less than 5% error compared to exact methods.
  • High-Dimensional Performance: FALCON successfully estimates LCDs for a 77-dimensional protein expression dataset, a feat impossible for prior methods.
  • Missing Data Imputation: On the protein dataset, FALCON outperforms other imputation methods (MVI, MGI, ICE, KNN) in terms of Root Mean Squared Error (RMSE).
  • Uncertainty Quantification: FALCON provides better calibrated credible intervals for missing data compared to multivariate Gaussian imputation, showcasing its superior density estimation flexibility.

Limitations and Future Work

  • Non-Convex Optimization: While FALCON achieves good results, the training objective is non-convex, meaning it might not find the absolute global optimum. The paper notes that experiments suggest it learns LCDs "close to the MLE."
  • Convex Hull Constraint (Simulations): For simulated data, FALCON's energy function was constrained to be non-zero only over the convex hull of the data. This was computationally feasible for low dimensions but not for the high-dimensional real dataset, where the constraint was relaxed. Future work could explore more efficient ways to handle this constraint in high dimensions.
  • Generalization of EBMs: The paper suggests that this work contributes to understanding the expressive power of EBMs and how they can be improved for density estimation tasks. Further exploration into different EBM architectures and training strategies for constrained density estimation is a natural next step.

Summary updated

Abstract

A classic problem within statistics is log-concave density (LCD) estimation, which asks for the best log-concave density that maximizes the probability of observing some input data points. Due to the non-parametric nature of this problem, current algorithms are too computationally demanding to work beyond a few (i.e. ten) dimensions. We introduce a new approach that employs energy-based neural networks to convert the non-parametric problem into a parametric one over the network weights, enabling scalable LCD estimation in high dimensions. By leveraging deep learning infrastructure (e.g. GPUs), our method can learn LCDs up to thousands of times faster than existing approaches, while requiring hundreds of times fewer parameters. We further show that our method learns informative LCDs on a real protein expression dataset with 77 dimensions, which is beyond the capabilities of current LCD estimation algorithms.
BibTeX
@inproceedings{lin2023how,
  author = {Alexander Lin and Demba E. Ba},
  title = {{How to Train Your FALCON: Learning Log-Concave Densities with Energy-Based Neural Networks}},
  booktitle = {AABI},
  year = {2023},
  url = {https://openreview.net/forum?id=UP1r5-t-ov},
}
Figure