TL;DR
This paper introduces a novel graphical model representation for ARMA time-series models, enabling the use of standard EM algorithms for parameter estimation. By introducing a small Gaussian noise component, they create a "stochastic ARMA" (σARMA) model that not only allows for EM but also surprisingly improves prediction accuracy through better smoothing.
What Problem Does This Paper Address?
Traditional Autoregressive Moving Average (ARMA) time-series models are powerful tools for analyzing sequential data. However, their inherent deterministic relationships make them difficult to work with using standard machine learning algorithms like the Expectation-Maximization (EM) algorithm, which is crucial for parameter estimation, especially with missing data. This limitation hinders the application of graphical model inference techniques to ARMA models. Furthermore, incorporating external information (cross-predictors) into ARMA models in a principled way has been challenging.
Methods
The core contribution is the representation of ARMA models as directed graphical models. To overcome the deterministic nature that prevents EM algorithm application, the authors introduce a "stochastic ARMA" (σARMA) model. This is achieved by replacing the deterministic components of the ARMA model with Gaussian distributions having a small variance (σ). This modification allows the EM algorithm to be effectively used for parameter estimation.
They further extend this to σARMAXP (stochastic ARMA with cross-predictors), which allows for:
- Multiple time series with potentially different AR and MA orders.
- Selective inclusion of cross-predictor time series.
The EM algorithm is then adapted to estimate the parameters of these σARMAXP models, even in the presence of missing data. The paper also details how to perform one-step and multi-step forecasting using these models.
Datasets
The paper evaluates their methods using two collections of real-world economic data:
- US-Econo: Monthly data on 50 economic indicators from February 1992 to December 2002.
- JPN-Econo: Monthly data on 53 economic indicators from various sources, with lengths varying from 52 to 259 periods.
For evaluation, they created incomplete versions of these datasets by introducing randomly missing observations (10% to 50%) and also generated "filled-in" datasets using linear interpolation/extrapolation for comparison.
Key Insights and Ideas
- Deterministic to Stochastic for EM: The key insight is that by introducing a small amount of Gaussian noise (σ), a deterministic ARMA model can be transformed into a stochastic one (σARMA) that is amenable to EM-based parameter estimation. This is a crucial step for leveraging graphical model inference.
- Smoothing as a Side Effect: Surprisingly, the introduction of this small noise component not only enables EM but also leads to improved prediction accuracy through a "smoothing" effect. This suggests that stochasticity can regularize the model and lead to more robust forecasts.
- Principled Handling of Missing Data: The σARMA approach provides a theoretically sound way to handle missing data within the EM framework, outperforming simple interpolation methods.
- Flexible Cross-Prediction: The σARMAXP model offers a more general framework for incorporating cross-predictors compared to existing vector ARMA models, allowing for selective inclusion and different model orders for each time series.
Key Results
- Improved Accuracy: σARMA and σARMA* models consistently outperformed standard ARMA models in terms of predictive accuracy (measured by sequential predictive scores) on both US-Econo and JPN-Econo datasets. The differences were statistically significant.
- Effective Missing Data Handling: σARMA models trained on incomplete data (using EM) yielded significantly more accurate predictions than those trained on data filled in by interpolation, especially as the percentage of missing data increased.
- Cross-Predictor Benefits: The inclusion of cross-predictors in σARMAXP and σARMA*XP models led to slight but significant improvements in predictive accuracy for both datasets.
- Smoothing Effect: The paper demonstrates that the stochastic nature of σARMA models provides a beneficial smoothing effect, leading to better forecasts.
Limitations and Future Work
- Convergence Rate with Small σ: The paper notes that the EM algorithm's convergence rate decreases as σ approaches zero. For σ=0, the EM algorithm cannot improve initial parameter settings.
- Free Variance (σ): While the paper defines σARMA* with a free variance parameter, preliminary experiments suggest that this variance quickly converges to zero, limiting further parameter improvement. The authors intend to investigate this behavior further.
- Future Work: The authors suggest exploring the theoretical explanation for the observed behavior of the free variance parameter and further investigating the relationship between their graphical model inference and state-space models (like Kalman filtering). They also mention that their approach can be extended to ARIMA models.
Summary updated