PYLFIRE: Python implementation of likelihood-free inference by ratio estimation

Jan Kokko, Ulpu Remes, Owen Thomas, Henri Pesonen, Jukka CoranderWellcome 2019 · DOI

TL;DR

This paper introduces PYLFIRE, a Python package that makes it easier to perform "likelihood-free inference" (LFI) using a method called LFIRE. LFIRE is particularly useful when you have a complex computer model but can't easily calculate the probability of observing your data given the model's parameters. PYLFIRE streamlines this process, offering a user-friendly tool for researchers in fields like genetics and astronomy.

What Problem Does This Paper Address?

Many scientific fields rely on complex computer simulations to model phenomena, from the evolution of populations to the formation of galaxies. However, a major hurdle in using these models for statistical inference is the "likelihood function." This function tells us how probable our observed data is given a specific set of model parameters. For many complex simulators, calculating this likelihood is computationally intractable or impossible. This is where "likelihood-free inference" (LFI) methods come in. They aim to estimate model parameters and their uncertainties without directly using the likelihood function, often by relying on summary statistics of the data and simulations. The paper addresses the need for accessible and efficient LFI tools, specifically by implementing the LFIRE method.

Methods

The core of this paper is the introduction of PYLFIRE, a Python package that implements the LFIRE (Likelihood-Free Inference by Ratio Estimation) method. LFIRE tackles the LFI problem by reframing it as a density-ratio estimation task. Here's how it works:

  • Density-Ratio Estimation: Instead of directly estimating the likelihood, LFIRE models the ratio between the "data generating distribution" (what the observed data looks like) and the "marginal distribution" (what simulated data looks like when averaged over all possible parameter values from the prior).
  • Logistic Regression: This density ratio is modeled using logistic regression. The input to the logistic regression is a set of "summary statistics" extracted from the observed and simulated data.
  • Lasso Regularization: A key feature is the use of penalized logistic regression, specifically with the Lasso (Least Absolute Shrinkage and Selection Operator) penalty. This has a crucial side effect: it automatically performs "summary statistics selection." If a summary statistic doesn't contribute much to estimating the density ratio, its corresponding coefficient in the logistic regression model is shrunk to zero, effectively discarding it. This helps in identifying the most informative summary statistics without manual effort.
  • Integration with ELFI: PYLFIRE is built as an extension of the ELFI (Engine for Likelihood-Free Inference) software, a general-purpose LFI framework. This allows PYLFIRE to leverage ELFI's infrastructure for model definition, simulation, and parallelization.
  • Performance: The implementation utilizes efficient Fortran subroutines (via the glmnet package) for the logistic regression fitting and supports parallel computation to speed up the process.

Datasets

The paper does not introduce new benchmark datasets. Instead, it uses a simulated dataset from an ARCH(1) model for demonstration purposes. This model is a standard choice for time series analysis and is used to illustrate how PYLFIRE can estimate parameters in a relatively simple but non-trivial scenario. The observed data in this example is generated from a specific instance of the ARCH(1) model with known parameters, and PYLFIRE is used to recover these parameters.

Key Insights and Ideas

  • Automatic Summary Statistics Selection: The most significant insight is how Lasso regularization in logistic regression elegantly handles the often-difficult task of selecting informative summary statistics. This automates a crucial step in LFI, making the process more robust and less reliant on expert knowledge.
  • Bridging LFI and Classification: LFIRE cleverly reframes a statistical inference problem into a machine learning classification problem, leveraging powerful existing tools like logistic regression.
  • Efficiency through Fortran and Parallelization: The implementation's focus on performance by using optimized Fortran code and parallel processing is a key practical contribution, making LFI more feasible for larger or more complex models.
  • Extending ELFI's Capabilities: PYLFIRE's integration into the ELFI ecosystem broadens the range of LFI methods available to users, promoting a more comprehensive toolkit.

Key Results

The paper demonstrates PYLFIRE's functionality using an ARCH(1) time series model.

  • Parameter Recovery: PYLFIRE successfully estimates the posterior distributions of the model parameters ($\theta_1$ and $\theta_2$) for the ARCH(1) model. The estimated posterior means and MAP estimates are reported and appear to be close to the true simulated values.
  • Robustness to Irrelevant Statistics: A key demonstration shows that even when the model's summary statistics are augmented with irrelevant "white-noise" variables, PYLFIRE, due to its automatic summary statistics selection, still produces similar and accurate posterior estimates. This highlights the robustness of the LFIRE method and PYLFIRE's implementation.
  • Visualization Tools: PYLFIRE includes built-in functions for plotting marginal and pairwise posterior distributions, which are essential for interpreting the results.

Limitations and Future Work

  • Self-Containment: Reviewers noted that while the paper focuses on computational aspects, it could benefit from being more self-contained. This includes providing a more explicit pseudo-algorithm for the LFIRE method and a clearer distinction between observed and synthetic data in the introduction.
  • Guidance on Applicability: More guidance is needed on when the LFIRE method is most suitable, particularly concerning the number of parameters and the diffuseness of prior distributions relative to posteriors.
  • Comparison with Alternatives: A deeper comparison with other Python LFI packages, such as ABCpy, including numerical simulations, would be beneficial to highlight PYLFIRE's relative merits and deficiencies.
  • MAP Estimation: The paper could explore the feasibility of embedding PYLFIRE's functionalities within an optimization process to directly find the Maximum A Posteriori (MAP) estimate, especially when estimating the entire posterior distribution is challenging.
  • User-Defined Classifiers: Allowing users to provide custom Python functions for classification (beyond the default logistic regression) could enhance flexibility and leverage advances in classification algorithms.
  • Performance Gains: While performance is a stated goal, the paper could benefit from more explicit investigations into the performance gains achieved through parallelization and optimized code, perhaps with more complex models or higher-dimensional parameter spaces.

Summary updated

Abstract

<ns4:p>Likelihood-free inference for simulator-based models is an emerging methodological branch of statistics which has attracted considerable attention in applications across diverse fields such as population genetics, astronomy and economics. Recently, the power of statistical classifiers has been harnessed in likelihood-free inference to obtain either point estimates or even posterior distributions of model parameters. Here we introduce PYLFIRE, an open-source Python implementation of the inference method LFIRE (likelihood-free inference by ratio estimation) that uses penalised logistic regression. PYLFIRE is made available as part of the general ELFI inference software http://elfi.ai to benefit both the user and developer communities for likelihood-free inference.</ns4:p>
BibTeX
@inproceedings{kokko2019pylfire,
  author = {Jan Kokko and Ulpu Remes and Owen Thomas and Henri Pesonen and Jukka Corander},
  title = {{PYLFIRE: Python implementation of likelihood-free inference by ratio estimation}},
  booktitle = {Wellcome},
  year = {2019},
  doi = {10.12688/wellcomeopenres.15583.1},
}
Figure