Joint Probability Distribution of Prediction Errors of ARIMA

Xin Qin, Jyotirmoy V. DeshmukharXiv 2018 · OpenReview

TL;DR

This paper presents a novel method for calculating the joint probability distribution of prediction errors for multiple future time steps in ARIMA models. This is crucial for providing probabilistic guarantees in time series forecasting, especially for applications requiring robust monitoring and decision-making under uncertainty.

What Problem Does This Paper Address?

In time series analysis, particularly with models like ARIMA, predictions are often made for individual future time points. However, many real-world applications, such as system monitoring or control, require understanding the joint uncertainty across multiple future time steps. For instance, knowing the probability that a signal will stay within a certain range for the next five minutes is more valuable than just knowing the probability for each individual minute. Existing methods often provide point predictions or individual error distributions, but lack a framework for quantifying the correlated uncertainty across a sequence of predictions. This paper aims to fill that gap by deriving the joint probability distribution of prediction errors for multiple future time steps in ARIMA models, covering both stationary and intrinsically stationary processes, and extending the analysis to both univariate and multivariate time series.

Methods

The core of the paper's methodology lies in deriving the mathematical expressions for the joint probability distribution of prediction errors. This involves:

  • Defining Prediction Error: The error for a future time step $n+h$ is defined as $Err(X_{n+h}) = P_n X_{n+h} - X_{n+h}$, where $P_n X_{n+h}$ is the best linear predictor of $X_{n+h}$ based on observations up to time $n$.
  • Leveraging ARIMA Properties: The derivation heavily relies on the properties of ARIMA models, particularly their representation as linear combinations of white noise.
  • Multivariate Normal Distribution: The paper establishes that prediction errors, under certain conditions (especially for stationary processes), follow a multivariate normal distribution. This allows for the characterization of their joint behavior using covariance matrices.
  • Matrix Algebra and Transformations: Extensive use of matrix operations is made to represent and manipulate the relationships between different time steps and their prediction errors. This includes defining transformation matrices that map white noise to prediction errors.
  • Handling Stationarity and Intrinsic Stationarity: The paper distinguishes between stationary processes and intrinsically stationary processes (where differencing makes them stationary). Different mathematical treatments are provided for each case, particularly in deriving the prediction error distributions.
  • Univariate and Multivariate Extensions: The framework is developed for univariate time series and then generalized to multivariate cases by extending the matrix operations and definitions to handle vector-valued time series and their corresponding covariance matrices.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Prediction Errors are Linearly Related to White Noise: A fundamental insight is that the prediction errors of ARIMA models, like the model's output itself, can be expressed as linear combinations of the underlying white noise process. This is key to their probabilistic behavior.
  • Joint Distribution of Errors is Multivariate Normal: For stationary ARIMA processes, the prediction errors across multiple future time steps are shown to follow a multivariate normal distribution. This is a powerful result as it allows for complete characterization of their joint uncertainty.
  • Importance of Differencing for Non-Stationary Cases: For intrinsically stationary processes, the paper demonstrates how differencing can be used to transform the problem into a stationary one, enabling the application of similar techniques.
  • Systematic Derivation for Univariate and Multivariate: The paper provides a structured approach to derive these distributions, first for univariate series and then extending it to multivariate series through careful matrix formulation.

Key Results

The paper's primary result is the derivation of the covariance matrices for the joint distribution of prediction errors for multiple future time steps in ARIMA models. Specifically:

  • Theorem 1 & 2: Provide the covariance matrices for prediction errors in univariate stationary and ARMA(p,q) processes, respectively.
  • Theorem 4: Extends the result to intrinsically stationary univariate processes, showing the joint distribution of errors is multivariate normal.
  • Theorem 5 & 6: Generalize the findings to multivariate stationary and ARMA(p,q) processes, providing their respective joint error distributions.
  • Theorem 7: Addresses the joint distribution of errors for multivariate intrinsically stationary processes.

While the paper focuses on the theoretical derivation of these distributions, it lays the groundwork for quantitative analysis by providing the exact covariance matrices.

Limitations and Future Work

  • Assumptions: The derivations rely on the standard assumptions of ARIMA models, including the linearity of the process and the properties of the white noise (e.g., zero mean, constant variance). The paper does not explore the impact of violations of these assumptions.
  • Computational Complexity: While the theoretical framework is established, the practical computation of these covariance matrices, especially for high-dimensional multivariate series or long prediction horizons, could be computationally intensive.
  • Model Identification: The paper assumes the ARIMA model parameters (p, q, and coefficients) are known. In practice, model identification is a crucial first step, and the impact of model identification errors on the derived joint distributions is not discussed.
  • Future Work: The paper implicitly suggests future work in applying these derived distributions for practical tasks like probabilistic forecasting, risk assessment, and robust control system design, especially in the context of Signal Temporal Logic (STL) as mentioned in the introduction.

Summary updated

Abstract

Producing probabilistic guarantee for several steps of a predicted signal follow a temporal logic defined behavior has its rising importance in monitoring. In this paper, we derive a method to compute the joint probability distribution of prediction errors of multiple steps based on Autoregressive Integrated Moving Average(ARIMA) model. We cover scenarios in stationary process and intrinsically stationary process for univariate and multivariate.
BibTeX
@misc{qin2018joint,
  author = {Xin Qin and Jyotirmoy V. Deshmukh},
  title = {{Joint Probability Distribution of Prediction Errors of ARIMA}},
  year = {2018},
  eprint = {1811.04685},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.04685},
}
Figure