ARMA Time-Series Modeling with Graphical Models

Bo Thiesson, David Maxwell Chickering, David Heckerman, Christopher Meek

TL;DR

This paper reimagines the classic Autoregressive Moving Average (ARMA) time-series model as a directed graphical model. By replacing the model's strict deterministic relationships with a stochastic approximation (Gaussian noise with a tiny variance), the authors unlock the use of the Expectation-Maximization (EM) algorithm. This novel framework, called stochastic ARMA ($\sigma$ARMA), elegantly handles missing data, seamlessly integrates cross-predictors from multiple time series, and surprisingly improves forecasting accuracy through superior smoothing.

What Problem Does This Paper Address?

Time-series forecasting often relies on ARMA models, but standard estimation techniques struggle when data is missing or when trying to incorporate "cross-predictors"—external, non-temporal covariates or related time series.

While graphical models (like Bayesian networks) are excellent at handling missing data and multi-series dependencies, mapping a classic ARMA model directly into a graphical structure introduces a major roadblock: deterministic nodes. In a standard ARMA model, the current observation is a deterministic function of past observations and past errors. In a graphical model, these rigid relationships cause the probability distributions to collapse, making it mathematically impossible for the standard Expectation-Maximization (EM) algorithm to learn or update the model's parameters.

The authors set out to bridge this gap, creating a flexible, graphical representation of ARMA that retains the power of EM learning.

Methods

The authors introduce Stochastic ARMA ($\sigma$ARMA). The core innovation is simple yet powerful: replace the deterministic equation of ARMA with a conditional Normal distribution:

$$Y_t | \text{past variables} \sim N(\mu, \sigma)$$

where $\sigma$ is a small, user-defined variance acting as a "stochastic relaxation." When $\sigma \to 0$, the model converges back to a classic deterministic ARMA model. However, keeping $\sigma > 0$ ensures the probability space doesn't collapse, allowing the EM algorithm to function smoothly.

ARMA(2,2) model represented as a directed graph. Stochastic nodes are single circles; deterministic nodes are double circles.

Using this graphical foundation, they extend the model to:

  1. $\sigma$ARMA$^*$: Allows the moving average impact parameter ($\beta_0$) to vary freely.
  2. $\sigma$ARMA$^{xp}$: Integrates selective cross-predictors ($C_t$) from other related time series directly into the mean function ($\mu$), offering a more flexible alternative to traditional vector ARMA models.

Parameter estimation is handled via standard junction-tree inference algorithms for Gaussian Bayesian networks during the E-step, paired with pseudo-inversion to solve the M-step linear equations safely, even when matrices are near-singular.

Datasets

The authors evaluated their approach using two real-world macroeconomic datasets compiled by Economagic:

  • US-Econo: 50 monthly economic indicators from February 1992 to December 2002 (all series are of length 131).
  • JPN-Econo: 53 monthly economic indicators from January 1981 to July 2002 (lengths vary from 52 to 259).

To test robustness against missing data, they generated incomplete subsets by randomly removing 10%, 20%, 30%, 40%, and 50% of the observations, comparing their EM-based approach against a standard heuristic where missing values are filled via linear interpolation.

Key Insights and Ideas

  • The Power of "Softening" Constraints: Injecting a tiny amount of variance ($\sigma$) into a deterministic relationship is a highly effective trick. It acts as a gateway to using robust probabilistic tools (like EM) on models that were previously mathematically locked.
  • Principled Smoothing Beats Ad-Hoc Tweaks: Traditionally, researchers might smooth an ARMA model by manually adding variance to the final prediction. The authors show that embedding the noise directly into the model structure ($\sigma$ARMA) yields significantly better, more mathematically sound smoothing.
  • Unified Architecture for Complex Forecasting: By framing time series as a Bayes net, handling missing data and adding cross-predictors become structural properties of the graph rather than separate engineering challenges (like those found in complex state-space/Kalman filter setups).

Key Results

The quality of the models was evaluated using the sequential predictive score (average log-likelihood of one-step-ahead forecasts on a 12-month holdout set).

  • Smoothing Performance: On complete datasets, $\sigma$ARMA and $\sigma$ARMA$^*$ significantly outperformed both standard ARMA and ad-hoc smoothed ARMA. For instance, on the US-Econo dataset, the average score improved from -4.928 (standard ARMA) to -4.520 ($\sigma$ARMA).
  • Handling Missing Data: When data was missing, using EM to learn directly from the incomplete data consistently beat the common practice of linear interpolation. This advantage became pronounced and statistically significant at 30% or more missing data.
Figure 3: Average sequential predictive scores for com- plete and incomplete data collections.
  • Cross-Prediction Success: Introducing cross-predictors ($\sigma$ARMA$^{xp}$) provided an additional boost to predictive accuracy across both datasets.

Limitations and Future Work

While highly effective, the approach has a few notable limitations:

  • The $\sigma$ Dilemma: The convergence rate of the EM algorithm slows down drastically as $\sigma$ approaches 0.
  • Static Variance: Ideally, $\sigma$ would be a free parameter learned from the data. However, the authors noted that when $\sigma$ is allowed to vary freely, it quickly drives itself to 0 during optimization, which breaks the EM algorithm all over again. A theoretical explanation for this behavior remains open.
  • Kalman Filter Equivalence: The authors note that a single filtering step in a Kalman filter is structurally equivalent to forward propagation in their clique representation. Formally proving this exact equivalence for Kalman smoothing and back-propagation is flagged for future work.

Summary updated

Abstract

We express the classic ARMA time-series model as a directed graphical model. In doing so, we find that the deterministic relationships in the model make it effectively impossible to use the EM algorithm for learning model parameters. To remedy this problem, we replace the deterministic relationships with Gaussian distributions having a small variance, yielding the stochastic ARMA (ARMA) model. This modification allows us to use the EM algorithm to learn parmeters and to forecast,even in situations where some data is missing. This modification, in conjunction with the graphicalmodel approach, also allows us to include cross predictors in situations where there are multiple times series and/or additional nontemporal covariates. More surprising,experiments suggest that the move to stochastic ARMA yields improved accuracy through better smoothing. We demonstrate improvements afforded by cross prediction and better smoothing on real data.
BibTeX
@inproceedings{thiesson2004arma,
  author = {Bo Thiesson and David Maxwell Chickering and David Heckerman and Christopher Meek},
  title = {{ARMA Time-Series Modeling with Graphical Models}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2004},
}
Figure