Introduction
Probabilistic Programming Languages (PPLs) allow researchers to express complex statistical models as code. However, a major technical hurdle remains: discreteness. Modern inference algorithms, such as Hamiltonian Monte Carlo (HMC) and Variational Inference (VI), are "gradient-based," meaning they require the model to be continuous and differentiable. When a program contains discrete distributions (like Poisson or Binomial) or "if-then" branches based on discrete values, these high-performance algorithms often fail or become prohibitively slow. Traditionally, researchers had to manually rewrite their models into continuous approximations—a task requiring deep expertise and prone to error.
The Leios Framework
To address this, the authors introduce Leios, an automated framework that transforms probabilistic programs containing discrete or hybrid variables into fully continuous versions. This "continualization" process allows models to leverage modern inference engines without requiring manual intervention.
The Leios pipeline consists of two primary stages:
- Program Transformation: Discrete distributions are replaced with continuous counterparts (e.g., replacing a Poisson with a Gaussian or Gamma) that match the original's first moments. Crucially, Leios performs "predicate correction," replacing exact equality checks (e.g.,
if x == 4) with optimized interval "holes" to maintain the program's logical flow in a continuous space. It also applies Gaussian smoothing to observed variables to ensure the entire likelihood function is differentiable. - Parameter Synthesis: The framework treats the newly introduced intervals as parameters to be optimized. It uses a training phase to find values that minimize the Wasserstein Distance (or "Earth Mover's Distance") between the original discrete model and the new continuous approximation. Unlike other metrics, Wasserstein distance remains stable when comparing discrete and continuous distributions, making it an ideal tool for this optimization.
Performance and Accuracy
Leios was evaluated against a suite of 10 benchmarks across fields like biology, cognitive science, and machine learning. The results highlight two major advantages:
- Speed: In WebPPL, Leios-transformed programs were approximately 4.8x faster on average than their discrete originals (0.584s vs. 2.797s).
- Accuracy: Leios consistently outperformed "naive" smoothing techniques. By correcting for control flow, it maintains a distribution that closely mirrors the original discrete logic, whereas naive methods often lead to significant inaccuracies or "starve" certain program branches.
Beyond speed and accuracy, Leios enables workflows in modern PPLs like Pyro that were previously impossible due to unsupported discrete latent variables. By providing formal guarantees of absolute continuity, Leios ensures that the resulting programs are mathematically compatible with gradient-based solvers.
Conclusion
Leios represents a significant step forward in making PPLs more accessible and efficient. By automating the transition from discrete to continuous models, it enables the use of powerful gradient-based solvers on a wider class of problems. While challenges remain in automatically tuning hyperparameters like the smoothing factor $\beta$ and handling extremely deep nested logic, Leios provides a robust foundation for high-performance probabilistic modeling.
Summary updated