A pipeline for systematic comparison of model levels and parameter inference settings applied to negative feedback gene regulation

Adrien Coulier, Prashant Singh, Marc Sturrock, Andreas HellanderbioRxiv 2021 · DOI

TL;DR

This paper introduces a computational pipeline to systematically evaluate how different model complexities, data amounts, and inference methods affect the accuracy of parameter estimation in gene regulatory networks. They demonstrate that a detailed spatial model, combined with a sophisticated distance metric, can lead to significantly better inference, even when observations are coarse.

What Problem Does This Paper Address?

When building computational models of biological systems like gene regulatory networks (GRNs), scientists face a crucial decision: how much detail (fidelity) should the model include? More detailed models can capture finer biological nuances but come with a massive computational cost for simulation and parameter inference. Simpler models are computationally cheaper but might miss important biological phenomena. The challenge is to determine which model fidelity is "just right" for a given question and available data, especially when the goal is to infer the true underlying physical parameters of the system. This is further complicated by the fact that standard inference methods, like Approximate Bayesian Computation (ABC), require many simulations and careful selection of summary statistics.

Methods

The authors developed a computational pipeline to tackle this problem. The core idea is to generate synthetic "ground truth" data using a high-fidelity spatial model (Smoldyn) across a wide range of parameter values. Then, they use this data to train Gaussian Processes (GPs) to approximate the "distance metric" – a function that quantifies how well simulated data matches observed data. This GP-based distance metric allows for much faster parameter inference using ABC (specifically, ABC-SMC) without needing to run expensive simulations repeatedly. The pipeline systematically compares different model levels (a detailed spatial model, a coarse-grained multiscale model, and a simple well-mixed model), various data scenarios (e.g., different amounts of data), and different distance metrics (naive summary statistics, optimized summary statistics, and the Kolmogorov distance).

Datasets

The study uses synthetic data generated from a detailed spatial model (Smoldyn) of a negative feedback gene regulation network (Hes1 system). This synthetic data serves as the "ground truth" for evaluating parameter inference accuracy across different model fidelities and inference settings. The data consists of simulated trajectories of protein and mRNA copy numbers.

Key Insights and Ideas

  • High-fidelity models can be justified even with coarse data: The study shows that a detailed spatial model (Smoldyn) can yield significantly better inference results than simpler models, even when the data is coarse (e.g., time series sampled at infrequent intervals). This suggests that the underlying spatial dynamics can be implicitly captured and leveraged by sophisticated inference methods.
  • The choice of distance metric is critical: The Kolmogorov distance, which compares distributions of molecular counts, often outperforms simpler summary statistics, especially when using the most detailed model. This highlights that how you compare your model's output to the data can be as important as the model itself.
  • Model error can dominate: For simpler models (WMM, CBM), inference accuracy plateaus even with more data or better distance metrics, indicating that the model's inherent limitations are the bottleneck, not the data or inference method.
  • Protein data is more informative than mRNA data: For this specific system, measuring protein levels generally leads to more accurate inference than measuring only mRNA.
  • Computational cost is a major hurdle, but GPs offer a solution: The pipeline's reliance on Gaussian Processes to approximate the distance metric is key to making these systematic comparisons computationally feasible.

Key Results

  • The detailed spatial model (Smoldyn) consistently outperformed the well-mixed (WMM) and coarse-grained multiscale (CBM) models in parameter inference accuracy, particularly when using the Kolmogorov distance.
  • When using summary statistics, the CBM performed nearly as well as Smoldyn, suggesting it can capture essential spatial effects. However, with the Kolmogorov distance, Smoldyn showed a clear advantage.
  • Reducing the amount of temporal data (sampling frequency) had a minimal impact on inference accuracy for the WMM and CBM, indicating model error was the limiting factor. For Smoldyn, more data did lead to improved accuracy.
  • Measuring only mRNA levels resulted in the least accurate inference, while measuring proteins or both mRNA and proteins yielded similar, better results.

Limitations and Future Work

  • Identifiability: The paper acknowledges that proving parameter identifiability (whether parameters can be uniquely determined from data) is a hard problem, especially for complex models. Their approach focuses on quantifying inference error rather than proving identifiability.
  • "Perfect Synthetic Data": The study uses "perfect synthetic data" without measurement noise. The authors suggest that future work could incorporate measurement error models to assess the robustness of their findings.
  • Statistics Selection: While they used an automated method for selecting summary statistics, they note that it's a challenging problem and other sets of statistics might perform even better.
  • Future Directions: The authors propose using this pipeline as a preliminary step to guide modelers in choosing appropriate models, data types, and inference settings before embarking on costly experimental data collection and analysis. They also suggest exploring methods that automatically learn optimal statistics.

Summary updated

Abstract

Abstract Quantitative stochastic models of gene regulatory networks are important tools for studying cellular regulation. Such models can be formulated at many different levels of fidelity. A practical challenge is to determine what model fidelity to use in order to get accurate and representative results. The choice is important, because models of successively higher fidelity come at a rapidly increasing computational cost. In some situations, the level of detail is clearly motivated by the question under study. In many situations however, many model options could qualitatively agree with available data, depending on the amount of data and the nature of the observations. Here, an important distinction is whether we are interested in inferring the true (but unknown) physical parameters of the model or if it is sufficient to be able to capture and explain available data. The situation becomes complicated from a computational perspective because inference and model selection need to be approximate. Most often it is based on likelihood-free Approximate Bayesian Computation (ABC) and here determining which summary statistics to use, as well as how much data is needed to reach the desired level of accuracy, are difficult tasks. Ultimately, all of these aspects - the model fidelity, the available data, and the numerical choices for inference and model selection - interplay in a complex manner. In this paper we develop a computational pipeline designed to systematically evaluate inference accuracy for a wide range of true known parameters. We then use it to explore inference settings for negative feedback gene regulation. In particular, we compare a spatial stochastic model, a coarse-grained multiscale model, and a simple well-mixed model for several data-scenarios and for multiple numerical options for parameter inference. Practically speaking, this pipeline can be used as a preliminary step to guide modelers prior to gathering experimental data. By training Gaussian processes to approximate the distance metric, we are able to significantly reduce the computational cost of running the pipeline.
BibTeX
@misc{coulier2021pipeline,
  author = {Adrien Coulier and Prashant Singh and Marc Sturrock and Andreas Hellander},
  title = {{A pipeline for systematic comparison of model levels and parameter inference settings applied to negative feedback gene regulation}},
  year = {2021},
  doi = {10.1101/2021.05.16.444348},
  howpublished = {bioRxiv},
}
Figure