Time Series Prediction using Mixtures of Experts

Assaf J. Zeevi, Ron Meir, Robert J. AdlerNeurIPS 1996 · OpenReview

Introduction

Predicting the future values of a time series is a fundamental challenge across fields ranging from finance to engineering. Traditional approaches often rely on linear models like AutoRegressive Moving Average (ARMA) due to their mathematical tractability and ease of estimation. However, real-world data is frequently non-linear. While non-linear models such as Neural Networks (NNs) offer enhanced predictive power, they often act as "black boxes" that lack theoretical transparency.

This paper bridges that gap by extending the Mixture of Experts (MEM) architecture to stationary time series prediction. The authors ask: Can we build a model as powerful as a neural network but constructed from the familiar, tractable components of linear autoregressive models? By providing a rigorous theoretical foundation, the authors prove that a mixture of simple linear autoregressive models can act as a universal approximator for complex non-linear processes.

Methodology: The Mixture of Experts Framework

The core of the proposed approach is the Mixture of Experts (MEM) framework, which integrates two primary components:

  1. Expert Networks: A collection of $n$ local models. In this context, each expert is a simple Linear Autoregressive (AR) model of order $d$.
  2. Gating Network: A non-linear function—specifically, a multiple-output logistic regression—that evaluates past data to determine which expert’s prediction should be weighted most heavily for the current time step.

The power of this architecture lies in its integration. While individual experts are linear, the gating network enables the global model to capture highly non-linear dynamics. Essentially, the model partitions the input space, assigning distinct linear models to different "regimes" of the time series.

Theoretical Foundations and Key Insights

The paper focuses heavily on theoretical proofs, establishing MEM on the same rigorous footing as feedforward neural networks. Key insights include:

  • Universal Approximation: The authors demonstrate that even if the individual experts are "degenerate" (reduced to simple constants), the mixture can approximate any smooth function in a Sobolev space.
  • The Role of $n$: The number of experts ($n$) in an MEM is mathematically analogous to the number of hidden units in a neural network. Increasing $n$ reduces approximation error but simultaneously increases the risk of overfitting (variance).
  • Bridging Statistical Traditions: The MEM framework is shown to be a generalization of established statistical models, such as Tong’s Threshold Autoregressive (TAR) model and Priestley’s State Dependent Models (SDM), providing a unified perspective on these disparate approaches.
  • Bias-Variance Tradeoff: The authors derive an explicit formula for the Mean Squared Error (MSE) that cleanly separates approximation error (bias from having too few experts) and estimation error (variance from having too many parameters relative to the dataset size $N$).

Main Results and Error Bounds

While empirical benchmarking is secondary to the theoretical derivations, the authors validated the model using synthetic data for known stochastic processes and real-world data to compare against standard models. The standout contribution is the "Main Theorem," which establishes upper bounds on estimator risk:

  • Convergence Rate: The approximation error decreases at a rate proportional to $n^{-2r/d}$, where $r$ represents the smoothness of the target function and $d$ is the lag size.
  • Total MSE Bound: For a sufficiently large dataset of size $N$, the total error is bounded by: $$\text{MSE} \leq \frac{c}{n^{2r/d}} + \frac{m_n^*}{2N} + o\left(\frac{1}{N}\right)$$

This bound is highly practical, allowing researchers to perform principled model selection (choosing the optimal number of experts $n$) by minimizing this upper bound, akin to using AIC or MDL criteria. In numerical tests, the MEM compared very favorably to ARMA, TAR, and standard neural networks without requiring extensive fine-tuning.

Limitations and Future Directions

Despite its theoretical strengths, the proposed MEM framework has a few limitations:

  • Known Lag Size: The theoretical derivations assume the lag size $d$ (how far back in time the model looks) is known a priori. In practice, determining the optimal lag is a non-trivial challenge.
  • Stationarity Assumptions: The proofs rely on the assumption that the time series is "strictly stationary" and "exponentially $\alpha$-mixing" (meaning the process's memory of the past decays rapidly). Consequently, the model may struggle with non-stationary data featuring trends or shifting seasonalities without further modifications.
  • Generalization Proofs: The authors conjecture that MEM represents a more general class of models than feedforward neural networks, but note that a formal proof of this specific hierarchy remains an area for future work.

Summary updated

Abstract

We consider the problem of prediction of stationary time series, using the architecture known as mixtures of experts (MEM). Here we suggest a mixture which blends several autoregressive models. This study focuses on some theoretical foundations of the predic(cid:173) tion problem in this context. More precisely, it is demonstrated that this model is a universal approximator, with respect to learn(cid:173) ing the unknown prediction function . This statement is strength(cid:173) ened as upper bounds on the mean squared error are established. Based on these results it is possible to compare the MEM to other families of models (e.g., neural networks and state dependent mod(cid:173) els). It is shown that a degenerate version of the MEM is in fact equivalent to a neural network, and the number of experts in the architecture plays a similar role to the number of hidden units in the latter model.
BibTeX
@inproceedings{zeevi1996time,
  author = {Assaf J. Zeevi and Ron Meir and Robert J. Adler},
  title = {{Time Series Prediction using Mixtures of Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1996},
  url = {https://openreview.net/forum?id=rkWyQw-_ZB},
}
Figure