Distributions.jl: Definition and Modeling of Probability Distributions in the JuliaStats Ecosystem

Mathieu Besançon, David Anthoff, Alex Arslan, Simon Byrne, Dahua Lin, Theodore Papamarkou, John PearsonJ. Stat. Softw 2019 · DOI · OpenReview

TL;DR

This paper introduces Distributions.jl, a comprehensive Julia package for defining, modeling, and sampling from probability distributions. It leverages Julia's powerful type system and multiple dispatch to provide an efficient, flexible, and unified interface for a wide range of statistical tasks, from basic density estimation to complex probabilistic programming.

What Problem Does This Paper Address?

Statistical modeling and analysis heavily rely on the concept of probability distributions. However, implementing these distributions and their associated operations (like sampling, density evaluation, and fitting) can be complex and error-prone. Existing libraries often have limitations in terms of flexibility, efficiency, or integration with other tools. The authors aim to address this by creating a robust, high-performance, and user-friendly package for probability distributions within the Julia ecosystem, making advanced statistical computations more accessible.

Methods

The core of Distributions.jl is built upon Julia's advanced type system and multiple dispatch. Key methodological contributions include:

  • Abstract Type Hierarchy: Distributions are organized into an abstract type hierarchy (Distribution{VariateForm, ValueSupport}) that captures properties like dimensionality (univariate, multivariate, matrix-variate) and support type (discrete, continuous). This allows for generic implementations and extensions.
  • Sampleable Interface: A Sampleable abstract type is introduced for objects from which random samples can be drawn, decoupling sampling from the full definition of a distribution.
  • Generic Interface: A consistent interface for common operations like pdf, cdf, quantile, and rand is provided, enabling easy integration and extension.
  • Parametric Typing: Distributions are often defined as parametric types, allowing for flexibility in the underlying numeric types (e.g., Float64, BigFloat, automatic differentiation types).
  • Extensibility: The design emphasizes extensibility, allowing users to define new distributions, samplers, and even new value support or variate form types without modifying the core package.
  • Probabilistic Programming Integration: The package is designed to seamlessly integrate with probabilistic programming languages (PPLs) built in Julia, providing a common foundation for sampling and distribution representation.

Datasets

Not discussed in the paper. The paper focuses on the package's design and functionality, using synthetic data and examples for demonstration rather than specific datasets for training or evaluation.

Key Insights and Ideas

  • Julia's Type System as a Superpower: The paper strongly highlights how Julia's type system and multiple dispatch are fundamental to creating a flexible, efficient, and extensible statistical library. This allows for "zero-cost abstractions" where complex mathematical concepts are mapped directly to types, enabling compile-time dispatch.
  • Unified Interface for Diverse Tasks: Distributions.jl provides a single, consistent interface for a wide array of tasks, from simple random number generation to fitting complex mixture models and supporting probabilistic programming. This reduces the cognitive load for users and developers.
  • Extensibility is Key: The design prioritizes allowing external developers to easily add new distributions and functionalities, fostering a rich ecosystem around the core package.
  • Bridging Theory and Practice: The package effectively bridges the gap between theoretical statistical concepts and practical implementation in a high-performance programming language.

Key Results

  • Comprehensive Distribution Support: The package implements a large number of common probability distributions, making them readily available for use.
  • Efficient Implementations: Leveraging Julia's performance, operations like PDF computation on mixtures are shown to be faster than manual implementations, especially for homogeneous component types. For example, PDF computation on mixtures is on average faster than manual computation, with ratios of 1.24x, 5.88x, and 1.28x for different mixture types.
  • Foundation for Ecosystem: Distributions.jl serves as a foundational package for other Julia statistical libraries, including those for kernel density estimation and probabilistic programming.
  • Enabling Probabilistic Programming: The package's design facilitates the development of PPLs in Julia, offering a unified way to handle random number generation and distribution representation.

Limitations and Future Work

  • Limitations:
    • The paper notes that for heterogeneous component types in mixtures, the compiler's inability to infer types can lead to dynamic dispatch, impacting performance.
  • While many common distributions are included, the implementation of maximum likelihood estimation for all distributions, including mixtures and matrix-variate distributions, is an ongoing effort.
  • Future Work:
    • Continued development of estimation methods for mixture models.
  • Improved multivariate and matrix-variate support.
  • Implementation of maximum likelihood estimation for a wider range of distributions.
  • Further exploration of performance optimizations, particularly for heterogeneous mixtures.

Summary updated

Abstract

Random variables and their distributions are a central part in many areas of statistical methods. The Distributions.jl package provides Julia users and developers tools for working with probability distributions, leveraging Julia features for their intuitive and flexible manipulation, while remaining highly efficient through zero-cost abstractions.
BibTeX
@article{besancon2019distributions,
  author = {Mathieu Besançon and David Anthoff and Alex Arslan and Simon Byrne and Dahua Lin and Theodore Papamarkou and John Pearson},
  title = {{Distributions.jl: Definition and Modeling of Probability Distributions in the JuliaStats Ecosystem}},
  journal = {J. Stat. Softw.},
  year = {2019},
  doi = {10.18637/jss.v098.i16},
}
Figure