Robust calibration of hierarchical population models for heterogeneous cell populations

Carolin Loos, Jan Hasenauer

Abstract

Cellular heterogeneity is known to have important effects on signal processing and cellular decision making. To understand these processes, multiple classes of mathematical models have been introduced. The hierarchical population model (HPM) builds a novel class which allows for the mechanistic description of heterogeneity and explicitly takes into account subpopulation structures. However, this model requires a parametric distribution assumption for the cell population and, so far, only the normal distribution has been employed. Here, we incorporate alternative distribution assumptions into the model, assess their robustness against outliers and evaluate their influence on the performance of model calibration in a simulation study and a real-world application example. We found that alternative distributions provide reliable parameter estimates even in the presence of outliers, and can in fact increase the convergence of model calibration.

Introduction: The Challenge of Cellular Heterogeneity

In biology, no two cells are exactly alike. This "cellular heterogeneity" is a fundamental feature of biological systems, enabling populations to survive fluctuating environments and execute complex collective decisions. To study this phenomenon, researchers employ hierarchical population models (HPMs). These models describe how different subpopulations of cells behave over time using mechanistic equations, capturing the distribution of cellular properties across a population.

However, a significant limitation has hampered the practical use of HPMs: they traditionally rely on the assumption that cell properties follow a normal (Gaussian) distribution. In experimental reality, single-cell data is rarely so well-behaved. Factors like "dropouts" (missing data points), "doublets" (two cells measured as one), and inherent biological skewness introduce outliers. Because the normal distribution has "thin tails," it is highly sensitive to these outliers, often leading to biased parameter estimates and failed model calibrations.

Methods: Beyond the Normal Distribution

To address these limitations, we generalized the HPM framework to incorporate more flexible distribution assumptions. We derived the mathematical foundations—specifically the likelihood functions and their gradients—necessary to utilize three alternative distributions:

  1. Multivariate Skew Normal: Designed to capture populations with asymmetric distributions.
  2. Multivariate Student’s t: A "heavy-tailed" distribution that is naturally robust to outliers. Unlike the normal distribution, the Student’s t does not "panic" when encountering data points far from the mean, treating them as unlikely but possible.
  3. Negative Binomial: Specifically tailored for discrete count data, which is common in gene expression studies.
Comparison of normal, skew normal, and Student’s t distributions used to model cell populations.

A critical component of our approach is the implementation of gradient-based calibration. By providing exact analytical gradients for these new likelihood functions, we enable optimization algorithms to navigate the parameter space far more efficiently than traditional derivative-free methods.

Evaluation and Datasets

We evaluated the performance of these robust distributions using both synthetic and experimental data:

  • Simulated Motifs: We generated 108 synthetic datasets based on three common biological motifs: a conversion process, two-stage gene expression, and a birth-death process. These datasets were intentionally corrupted with three outlier scenarios: Zeros (simulating dropouts), Doublets (simulating merged signals), and Uniform (simulating random background noise).
  • Experimental Data: We re-analyzed a real-world dataset of NGF-induced Erk1/2 signaling in primary sensory neurons, a pathway central to pain sensitization.

Key Results: Robustness and Efficiency

Our findings demonstrate that moving beyond the normal distribution significantly improves both the accuracy and the reliability of model calibration.

Resilience to Outliers

In simulations involving outliers, the Student’s t distribution consistently achieved the lowest Mean Squared Error (MSE). While the normal distribution attempted to shift its mean to accommodate outliers—thereby misrepresenting the bulk of the population—the Student’s t distribution maintained an accurate fit of the primary population by assigning outliers to its heavier tails.

Fitting performance of different distributions under various outlier scenarios (zeros, doublets, and uniform noise).

Improved Optimization Performance

Counter-intuitively, using more complex distributions like the Student’s t or skew normal often made the optimization process faster and more reliable. By better representing the underlying data structure, these distributions create a smoother "likelihood landscape," making it easier for optimization algorithms to converge on the global optimum. In the real-world Erk1/2 signaling application, the Student’s t distribution substantially outperformed the normal distribution in terms of convergence speed and the number of successful calibration starts.

Biological Relevance of Skewness

In the analysis of experimental Erk1/2 data, the skew normal distribution provided the best fit. This suggests that the "lopsidedness" observed in cell populations is often a genuine biological property rather than mere measurement noise—a property that standard normal distributions effectively erase.

Conclusion and Future Directions

This work provides a robust framework for the calibration of hierarchical population models. By replacing the rigid normal distribution assumption with flexible alternatives like the Student’s t and skew normal distributions, we can obtain reliable biological insights even from messy, outlier-prone single-cell data.

Future work will focus on disentangling biological variability from measurement noise through the use of mixture models. Additionally, we aim to extend the multivariate application of discrete distributions, such as the Negative Binomial, to better account for correlations between multiple genes or proteins within individual cells.

Summary updated

Abstract

Abstract Cellular heterogeneity is known to have important effects on signal processing and cellular decision making. To understand these processes, multiple classes of mathematical models have been introduced. The hierarchical population model builds a novel class which allows for the mechanistic description of heterogeneity and explicitly takes into account subpopulation structures. However, this model requires a parametric distribution assumption for the cell population and, so far, only the normal distribution has been employed. Here, we incorporate alternative distribution assumptions into the model, assess their robustness against outliers and evaluate their influence on the performance of model calibration in a simulation study and a real-world application example. We found that alternative distributions provide reliable parameter estimates even in the presence of outliers, and can in fact increase the convergence of model calibration. Highlights Generalizes hierarchical population model to various distribution assumptions Provides framework for efficient calibration of the hierarchical population model Simulation study and application to experimental data reveal improved robustness and optimization performance
BibTeX
@article{loos2019robust,
  author = {Carolin Loos and Jan Hasenauer},
  title = {{Robust calibration of hierarchical population models for heterogeneous cell populations}},
  journal = {Journal of Theoretical Biology},
  year = {2019},
  doi = {10.1016/j.jtbi.2019.110118},
}
Figure