Hierarchical Non-Stationary Temporal Gaussian Processes With $L^1$-Regularization

Zheng Zhao, Rui Gao, Simo SärkkäarXiv 2021 · DOI · arXiv

Abstract: This paper is concerned with regularized extensions of hierarchical non-stationary temporal Gaussian processes (NSGPs) in which the parameters (e.g., length-scale) are modeled as GPs. In particular, we consider two commonly used NSGP constructions which are based on explicitly constructed non-stationary covariance functions and stochastic differential equations, respectively. We extend these NSGPs by including $L^1$-regularization on the processes in order to induce sparseness. To solve the resulting regularized NSGP (R-NSGP) regression problem we develop a method based on the alternating direction method of multipliers (ADMM) and we also analyze its convergence properties theoretically. We also evaluate the performance of the proposed methods in simulated and real-world datasets.

Introduction

Standard Gaussian Processes (GPs) are a cornerstone of probabilistic modeling, yet they often struggle with signals that exhibit sudden jumps, sharp transitions, or varying levels of smoothness. This limitation stems from the assumption of stationarity—the idea that the signal's characteristics, such as its length-scale (smoothness) and magnitude (intensity), remain constant across the entire input space.

When a stationary GP encounters a signal with a sudden discontinuity, it typically fails in one of two ways: it either over-smooths the jump to maintain global consistency or over-fits to local noise in an attempt to capture the sharp transition. While Non-Stationary GPs (NSGPs) allow these parameters to vary, they often produce "blurry" transitions. This research introduces Regularized NSGPs (R-NSGPs), which incorporate $L^1$-regularization to induce sparsity, allowing parameters to remain constant most of the time and change abruptly only when the data demands it.

Methodology

The proposed R-NSGP framework utilizes a hierarchical structure consisting of three layers. The top layer represents the signal $f(t)$, while two hidden layers are GPs that control the time-varying length-scale $u^\ell(t)$ and magnitude $u^\sigma(t)$.

$L^1$-Regularization and Sparsity

By adding an $L^1$ penalty (akin to LASSO) to the hidden parameter processes, the model encourages "sparse" deviations from a baseline. When applied to the derivatives of these parameters (Total Variation regularization), it forces the parameters to be piecewise constant, which is ideal for capturing sudden regime shifts in the underlying process.

Optimization via ADMM

Because $L^1$ penalties are non-differentiable and the hierarchical model is non-convex, standard gradient-based optimization is insufficient. The authors developed an optimization routine based on the Alternating Direction Method of Multipliers (ADMM). This approach splits the complex global problem into smaller, more manageable sub-problems that can be solved iteratively.

Algorithm 1: Regularized batch NSGP with ADMM (R-NSGP ADMM)

Computational Efficiency

The paper explores two implementations:

  1. Batch Construction: Uses explicit covariance functions, providing high accuracy but with $O(T^3)$ complexity.
  2. State-Space Construction: Formulates the GPs as Stochastic Differential Equations (SDEs), reducing complexity to linear time ($O(T)$). This makes the model viable for long-duration time-series data.

Experimental Evaluation

The method was evaluated using both synthetic and real-world datasets:

  • Synthetic Rectangular Signal: A noisy, piecewise constant signal designed to test the model's ability to capture sharp jumps. The Regularized State-Space NSGP (R-SS-NSGP) achieved an RMSE of $1.63 \times 10^{-2}$, significantly outperforming both standard stationary GPs ($3.99 \times 10^{-2}$) and vanilla non-stationary GPs ($4.43 \times 10^{-2}$).
  • Human Motion Data: Real-world accelerometer data from clinical trials tracking a subject's movement. The R-NSGP successfully filtered out high-frequency artifacts and sensor noise that caused standard GPs to produce erratic, jittery estimates.

Key Insights and Discussion

The core innovation of this work is the realization that non-stationarity alone is insufficient for "jumpy" signals; sparsity is the necessary "secret sauce." By adding $L^1$ regularization, the model ignores noise in stable regions while reacting decisively to actual jumps. Furthermore, the transition from batch covariance matrices to a state-space SDE formulation allows for dramatic improvements in scalability without sacrificing mathematical rigor.

However, some challenges remain. The objective function is non-convex, meaning the ADMM solver is guaranteed only to find a local minimum, making initialization critical. Additionally, while the model provides a Maximum A Posteriori (MAP) estimate, full Bayesian uncertainty quantification for the state-space version remains computationally demanding.

Conclusion

This research provides a robust framework for modeling non-stationary signals with discontinuities. By combining hierarchical GPs with $L^1$-regularization and efficient ADMM-based optimization, the authors have created a tool that is both theoretically grounded and practically applicable to complex, real-world time-series analysis.

Summary updated

Abstract

This paper is concerned with regularized extensions of hierarchical non-stationary temporal Gaussian processes (NSGPs) in which the parameters (e.g., length-scale) are modeled as GPs. In particular, we consider two commonly used NSGP constructions which are based on explicitly constructed non-stationary covariance functions and stochastic differential equations, respectively. We extend these NSGPs by including $L^1$-regularization on the processes in order to induce sparseness. To solve the resulting regularized NSGP (R-NSGP) regression problem we develop a method based on the alternating direction method of multipliers (ADMM) and we also analyze its convergence properties theoretically. We also evaluate the performance of the proposed methods in simulated and real-world datasets.
BibTeX
@misc{zhao2021hierarchical,
  author = {Zheng Zhao and Rui Gao and Simo Särkkä},
  title = {{Hierarchical Non-Stationary Temporal Gaussian Processes With $L^1$-Regularization}},
  year = {2021},
  eprint = {2105.09695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.09695},
}
Figure