LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models

Yuan Zhou, Bradley J. Gram-Hansen, Tobias Kohn, Tom Rainforth, Hongseok Yang, Frank Wood

Abstract

We develop a new Low-level, First-order Probabilistic Programming Language (LF-PPL) suited for models containing a mix of continuous, discrete, and/or piecewise-continuous variables. The key success of LF-PPL is its ability to automatically identify and classify discontinuities in a model's density function. This enables advanced inference algorithms—specifically Discontinuous Hamiltonian Monte Carlo (DHMC)—to operate efficiently on models that would typically break standard gradient-based samplers.

Introduction

Modern probabilistic programming languages (PPLs) have revolutionized statistical modeling by allowing researchers to define complex generative processes. However, a significant bottleneck remains: many powerful inference algorithms, such as Hamiltonian Monte Carlo (HMC), rely on the differentiability of the target density. When models include discrete variables or conditional logic (e.g., if-else branching), the resulting density becomes piecewise-continuous. In these scenarios, standard gradient-based methods often fail as they encounter "kinks" or "jumps" where gradient information is either missing or misleading.

We introduce LF-PPL, a low-level language and compiler designed to bridge this gap. By automatically detecting and classifying discontinuities, LF-PPL provides the necessary metadata to power specialized samplers like DHMC, which can navigate non-smooth landscapes with high efficiency.

The Challenge of Non-Smooth Models

In many real-world applications, models are not "smooth." Non-differentiability typically stems from branching logic where the choice of distribution depends on latent variables. Existing PPLs like Stan or Pyro do not expose the locations of these boundaries to the inference engine. Consequently, practitioners are often forced to manually rewrite models to "smooth out" discontinuities or resort to less efficient sampling methods that do not leverage gradient information.

Methodology: The LF-PPL Compiler

LF-PPL addresses this by focusing on a minimalistic, low-level syntax that serves as a compilation target for higher-level languages. The core of our approach lies in the compilation scheme:

  1. Linearized Intermediate Representation (LIR): The compiler transforms the model into an LIR that explicitly tracks variables involved in branching predicates.
  2. Automatic Classification: Latent variables are classified as "continuous" or "discontinuous" based on whether they affect the control flow of the program.
  3. Boundary Tracking: The compiler generates runtime checks to determine if a sampler has crossed a discontinuity boundary (i.e., changed a branch in an if statement).
  4. Mathematical Rigor: The authors provide a formal proof that models expressed in LF-PPL have densities where discontinuities occur on a set of measure zero, ensuring the mathematical validity of the underlying inference algorithms.

Experimental Evaluation

We evaluated LF-PPL using two primary benchmarks: a Gaussian Mixture Model (GMM) and a high-dimensional Heavy Tail Piecewise Model.

Automation vs. Manual Tuning

In the GMM experiment, we compared DHMC (running on LF-PPL) against the No-U-Turn Sampler (NUTS) in PyMC3. Despite DHMC being a relatively unoptimized implementation, the automated tracking of discontinuities allowed it to perform on par with the highly tuned NUTS sampler, proving that automated tracking is a viable path for general-purpose PPLs.

MSE vs. Sample Size for a Gaussian Mixture Model, comparing DHMC (via LF-PPL) to PyMC3's NUTS.

Scaling to High Dimensions

The advantages of LF-PPL are most apparent in high-dimensional non-smooth models. In the Heavy Tail Piecewise Model, standard HMC's performance degrades rapidly as dimensionality increases because the sampler frequently "trips" over boundaries. In contrast, DHMC—leveraging LF-PPL’s metadata to jump across boundaries—maintains high efficiency even at 500 dimensions.

Scaling performance at 500 dimensions: DHMC maintains efficiency while standard HMC fails to converge.

Conclusion and Future Work

LF-PPL demonstrates that low-level language design can unlock significant performance gains for non-differentiable probabilistic models. By automating the detection of discontinuities, we remove the need for manual model tuning and enable robust inference in complex, high-dimensional spaces. Future work includes extending LF-PPL to support Stochastic Variational Inference (SVI) and addressing challenges related to zero-density regions in the model space.

Summary updated

Abstract

We develop a new Low-level, First-order Probabilistic Programming Language (LF-PPL) suited for models containing a mix of continuous, discrete, and/or piecewise-continuous variables. The key succes...
BibTeX
@inproceedings{zhou2019lf,
  author = {Yuan Zhou and Bradley J. Gram-Hansen and Tobias Kohn and Tom Rainforth and Hongseok Yang and Frank Wood},
  title = {{LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2019},
  url = {https://openreview.net/forum?id=BDAi3y6p6ZD},
}
Figure