A Joint estimation approach to sparse additive ordinary differential equations

Nan Zhang, Muye Nanshan, Jiguo CaoStat. Comput 2022 · DOI

TL;DR

This paper introduces JADE, a novel method for jointly estimating parameters and latent states in sparse additive Ordinary Differential Equations (ODEs) with non-Gaussian data. JADE offers a unified framework that simultaneously considers data fidelity, ODE adherence, and sparsity, leading to more accurate parameter and state estimation, and improved network structure recovery.

What Problem Does This Paper Address?

Many real-world systems, from gene regulation to disease spread, are modeled using Ordinary Differential Equations (ODEs). However, estimating the parameters of these ODEs from noisy, often non-Gaussian, observational data is a significant challenge. Existing methods often fall into two categories: those that strictly enforce ODE compliance but are computationally intensive and numerically unstable, or those that use a two-stage approach where ODEs are used only as a discrepancy measure, potentially sacrificing accuracy. Furthermore, identifying the underlying sparse structure of these dynamic systems, crucial for interpretability, remains difficult. This paper aims to bridge this gap by proposing a unified approach that jointly estimates latent processes and ODE parameters while simultaneously recovering the sparse structure of the system, even with non-Gaussian observations.

Methods

The core of this paper is the Joint Estimation for Generalized Sparse Additive ODEs (JADE) approach. JADE formulates a single objective function that combines three key components:

  1. Data Fidelity: This term ensures the estimated latent processes (the unobserved states of the system) fit the observed data. It's framed within an exponential family distribution, allowing for non-Gaussian data.
  2. ODE Fidelity: This term penalizes deviations of the latent processes and their derivatives from the underlying ODE model, effectively regularizing the latent process estimation.
  3. Sparsity Regularization: A group lasso penalty is applied to the ODE parameters to encourage sparsity, meaning many of the regulatory effects are identified as zero, leading to a simpler and more interpretable model.

To tackle the non-convex and non-differentiable nature of the objective function, JADE employs a block coordinate descent algorithm. This algorithm iteratively updates blocks of parameters (latent process basis coefficients and ODE parameters) while holding others fixed. For updating latent processes, it uses a Newton-type step with a diagonal Hessian approximation. For updating ODE parameters, it uses an adaptive group lasso regression. The algorithm's global convergence is theoretically established.

Datasets

The paper evaluates JADE using simulated datasets generated from Gaussian, Poisson, and Bernoulli distributions. These simulations are designed to mimic real-world scenarios with varying sample sizes and signal-to-noise ratios.

For real-world applications, the paper uses:

  • Yeast Cell Cycle Gene Regulatory Network Data: This dataset, from Spellman et al. (1998), contains mRNA expression levels of 6,178 genes over time, used to infer regulatory relationships.
  • Stock Price Change Direction Data: This dataset comprises daily stock price changes for 40 companies across 8 sectors in 2020, modeled as Bernoulli observations to understand sector-level interactions.

Key Insights and Ideas

  • Unified Framework: JADE's strength lies in its unified approach, treating data fidelity, ODE adherence, and sparsity simultaneously. This contrasts with traditional two-stage methods and offers a more holistic estimation strategy.
  • Non-Gaussian Data Handling: The ability to handle non-Gaussian data through the exponential family framework significantly broadens the applicability of ODE modeling to a wider range of real-world phenomena.
  • ODE Fidelity as a Regularizer: Using the ODE fidelity term to regularize latent process estimation is a key innovation. It leverages the underlying system dynamics to improve the accuracy of both state and parameter estimates.
  • Block Coordinate Descent for Non-Convexity: The development of a block coordinate descent algorithm with provable convergence is crucial for optimizing the complex, non-convex objective function.
  • Network Recovery: The combination of ODE modeling and sparse regularization (group lasso) proves effective in identifying the underlying regulatory network structure, offering insights into system interactions.

Key Results

  • Improved Estimation Accuracy: JADE consistently outperforms competing methods (GRADE and SA-ODE) in estimating latent processes and ODE parameters, especially for non-Gaussian data and in scenarios with smaller sample sizes or lower signal-to-noise ratios. For instance, in Gaussian observations, JADE reduced MSE(θ) by up to 30% compared to other methods.
  • Effective Network Discovery: JADE demonstrates strong performance in identifying nonzero additive ODE components (network recovery), achieving high true-positive rates. While GRADE sometimes shows slightly better false-positive rates, JADE offers a more balanced performance across various metrics.
  • Robustness to Tuning Parameters: The study shows JADE is relatively insensitive to the ODE fidelity tuning parameter ($\lambda_\theta$) as long as it's not too small, simplifying practical implementation.
  • Real-World Application Success: JADE successfully identified key regulatory relationships in the yeast cell cycle GRN and revealed interesting nonlinear interactions between stock market sectors, aligning with known biological and economic phenomena.

Limitations and Future Work

  • Identifiability Issues: Due to the additive assumption in the ODE system, ODE parameters are not fully identifiable. The paper addresses this by imposing a mean-zero constraint on additive components and adjusting intercepts, but this is a known limitation of such models.
  • Computational Complexity: While the block coordinate descent algorithm is designed for efficiency, optimizing complex ODE systems can still be computationally demanding, especially for very high-dimensional systems.
  • Future Work: The paper suggests extending the approach to more complex ODE structures beyond additive models and exploring its application to other dynamic systems. Further investigation into the theoretical properties and practical implementation for even larger-scale problems would also be valuable.

Summary updated

BibTeX
@article{zhang2022joint,
  author = {Nan Zhang and Muye Nanshan and Jiguo Cao},
  title = {{A Joint estimation approach to sparse additive ordinary differential equations}},
  journal = {Stat. Comput},
  year = {2022},
  doi = {10.1007/s11222-022-10117-y},
}
Figure