System identification using Bayesian neural networks with nonparametric noise models

Christos Merkatas, Simo SärkkäarXiv 2021 · DOI · arXiv

Abstract

System identification is a cornerstone of science and engineering, yet traditional methods often rely on restrictive assumptions regarding functional forms and noise distributions. This paper presents a robust Bayesian nonparametric approach for identifying discrete-time nonlinear random dynamical systems. By integrating Bayesian Neural Networks (BNNs) for functional estimation with Geometric Stick Breaking (GSB) processes for noise modeling, the proposed framework—NP-BNN—obviates the need for Gaussian error assumptions. We demonstrate that as the number of hidden neurons increases, the model converges to a full nonparametric Bayesian regression. Using a customized Gibbs sampler incorporating Hamiltonian Monte Carlo, we show that NP-BNN significantly outperforms standard models on both chaotic simulated data and real-world ecological time series.

Introduction

In the study of dynamical systems, "system identification" refers to the process of inferring the underlying mathematical rules of a system from observed data. Historically, this field has leaned heavily on two simplifying assumptions: that the system follows a predefined functional form (e.g., linear or polynomial) and that the stochastic "noise" follows a Gaussian distribution.

However, real-world phenomena rarely adhere to these "bell curve" ideals. Environmental noise, measurement errors, and inherent system fluctuations are frequently skewed, multi-modal, or heavy-tailed. When a model assumes Gaussian noise but encounters heavy-tailed reality, it produces overconfident and often wildly inaccurate predictions. This paper addresses these shortcomings by introducing a "double nonparametric" framework that learns both the system dynamics and the noise structure directly from the data.

The NP-BNN Framework

The proposed NP-BNN model consists of two primary components designed to handle uncertainty and complexity simultaneously:

1. The Signal Engine: Bayesian Neural Networks

To model the deterministic part of the system, the authors utilize a Bayesian Neural Network. Unlike frequentist neural networks that assign fixed values to weights, BNNs treat weights as probability distributions. This allows for superior uncertainty quantification. A key theoretical contribution of this work is the highlight that as the hidden layer grows, the BNN effectively converges to a Gaussian Process (GP), providing a robust bridge to full nonparametric regression.

2. The Noise Engine: Geometric Stick Breaking (GSB)

Rather than assuming a single Gaussian distribution for error, the noise is modeled as an infinite mixture of Gaussians. This is achieved through a Geometric Stick Breaking (GSB) prior. GSB is a type of Bayesian nonparametric prior that allows the model to "learn" the specific shape of the noise. Whether the noise is a sharp peak or a flat, heavy-tailed distribution, the GSB process adds complexity only as the data demands it. The authors chose GSB over the more common Dirichlet Process (DP) because GSB weights are strictly decreasing, which facilitates faster and more efficient sampling for clustering.

Posterior Inference

To estimate the parameters of this complex model, the authors developed a customized Gibbs sampler. Within this framework, Hamiltonian Monte Carlo (HMC) is employed to navigate the high-dimensional weight space of the neural network, ensuring efficient convergence even in complex landscapes.

Experimental Results

The NP-BNN was evaluated against traditional models, including standard Artificial Neural Networks (ANNs) and polynomial-based models, across two distinct datasets:

Chaotic Logistic Map (Simulated)

To test the model's ability to recover signals from "dirty" data, the authors applied it to a noisy logistic map—a classic chaotic system. They introduced noise consisting of a mixture of two Gaussians with vastly different variances.

  • NP-BNN MSE: 0.1802
  • Standard BNN MSE: 0.2719
  • Polynomial Model MSE: 0.4324

The NP-BNN was the only model capable of accurately reconstructing the sharp peak of the true noise density, whereas standard models tended to over-smooth the distribution.

Canadian Lynx Data (Real-World)

This dataset, containing 114 years of annual lynx trappings, is a benchmark for nonlinear time series. The population cycles are notoriously difficult to predict due to their non-linear nature.

  • NP-BNN MSE: 0.0249
  • Standard ANN MSE: 0.0371
  • Classic AR(11) MSE: 0.0822

In terms of Mean Absolute Prediction Error (MAPE), the NP-BNN achieved approximately 4.27%, while the standard Gaussian-based BNN lagged behind at nearly 7%.

Discussion and Insights

The primary advantage of the NP-BNN lies in its flexibility and robustness. By using an infinite mixture for noise, the model avoids the common pitfall of overestimating noise variance. Standard models often stretch a single Gaussian to cover outliers, which degrades the accuracy of the "core" prediction. The NP-BNN avoids this by assigning outliers to secondary mixture components.

Furthermore, the fully Bayesian nature of the model provides a "cloud" of possible future states rather than a single point estimate. This is vital for risk assessment in engineering and ecological management, where understanding the range of possible outcomes is as important as the most likely outcome.

Limitations and Future Directions

One notable limitation is the non-reversibility of the model. In many dynamical systems, it is desirable to estimate the initial state ($y_0$) from current observations. Because the system function is modeled by a neural network, it is not easily invertible.

Future research may address this by incorporating Normalizing Flows—architectures specifically designed to be invertible—to allow for better estimation of past states. Additionally, extending this framework to continuous-time modeling via Neural Ordinary Differential Equations (Neural ODEs) represents a promising avenue for high-frequency physical systems.

Summary updated

Abstract

System identification is of special interest in science and engineering. This article is concerned with a system identification problem arising in stochastic dynamic systems, where the aim is to estimate the parameters of a system along with its unknown noise processes. In particular, we propose a Bayesian nonparametric approach for system identification in discrete time nonlinear random dynamical systems assuming only the order of the Markov process is known. The proposed method replaces the assumption of Gaussian distributed error components with a highly flexible family of probability density functions based on Bayesian nonparametric priors. Additionally, the functional form of the system is estimated by leveraging Bayesian neural networks which also leads to flexible uncertainty quantification. Asymptotically on the number of hidden neurons, the proposed model converges to full nonparametric Bayesian regression model. A Gibbs sampler for posterior inference is proposed and its effectiveness is illustrated on simulated and real time series.
BibTeX
@misc{merkatas2021system,
  author = {Christos Merkatas and Simo Särkkä},
  title = {{System identification using Bayesian neural networks with nonparametric noise models}},
  year = {2021},
  eprint = {2104.12119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.12119},
}
Figure