Combined Neural Networks for Time Series Analysis

Iris Ginzburg, David HornNeurIPS 1993

Instead of struggling to train a single, massive neural network for time series prediction, why not use two? This paper proposes a "coarse-to-fine" approach: train a primary network to capture the main signal, then train a secondary, simpler network to predict and correct the first one's errors. This two-step strategy significantly outperforms monolithic architectures and simplifies the training process.

The Challenge of Time Series Prediction

Predicting the next value in a time series—such as stock prices or weather patterns—is notoriously difficult because the data often contains both complex non-linear trends and systematic noise. In the early 1990s, the standard approach was to build increasingly complex neural networks to capture every detail.

However, the authors argue that "bigger is not always better." Large, complex networks are difficult to train, prone to getting stuck in poor local minima, and often fail to generalize. Looking to biology for inspiration, they note that the nervous system often processes information in stages, where later stages refine the work of earlier ones. The authors address how to implement this "refinement" logic in artificial neural networks to improve accuracy without the headaches of massive architectures.

The Combined Neural Network Approach

To tackle this, the authors introduce a Combined Neural Network consisting of two distinct stages:

  1. The Primary Network: A standard feed-forward network (using back-propagation) with one hidden layer of sigmoidal neurons and a linear output. It is trained on the raw time series data to produce an initial prediction ($p_n$).
  2. The Secondary Network: After the primary network is fully trained, the authors calculate the "residuals"—the errors between the prediction and the actual data ($q_n = y_n - p_n$). A secondary, simpler network (often just a linear perceptron) is then trained specifically to predict these residuals.
  3. The Combined Predictor: The final prediction is simply the sum of the two outputs: $t_n = p_n + r_n$.

Decoupling Training Rates

A major insight of the paper is that training everything at once in a single "Complex Network" often fails because different parts of the network require different learning speeds. If the linear parts train too fast, the non-linear parts never learn the underlying structure. Sequential training bypasses this "interference" entirely.

Comparison of prediction errors across different network architectures and learning rates.

As shown in the table above, attempting to train a single network with mixed learning rates for linear weights yields sub-optimal prediction errors ($\epsilon_p$). The sequentially trained "combined system" achieves the lowest prediction error, proving that decoupling the training stages is superior to simultaneous training.

Results on the Sunspots Dataset

The study evaluates this architecture using the Sunspots dataset, a classic benchmark consisting of yearly averages of sunspot activity (220 training points, 35 test points).

By treating a trained network’s errors not just as random noise but as systematic correlations the primary network missed, the secondary network effectively "uncovers and removes" these residual patterns. The authors view this secondary network as a first-order correction—much like a Taylor expansion—justifying why a simple linear model is often sufficient for the second stage.

  • Short-Term Accuracy: The combined system reduced training errors by 3–4% and improved test set predictions by more than 10% compared to the primary network alone.
  • Long-Term Accuracy: For multi-step-ahead predictions, the system feeds its corrected output back in as input, preventing the "error drift" that usually plagues long-term forecasts.
Multi-step prediction errors showing the combined network's superior long-term accuracy.

The table above illustrates multi-step prediction errors for 2, 5, and 11 years into the future ($\epsilon_2, \epsilon_5, \epsilon_{11}$). The primed columns ($\epsilon'_2, \epsilon'_5, \epsilon'_{11}$) represent the combined network, which consistently demonstrates lower error rates—reducing long-term forecasting errors by approximately 20%.

Limitations and Legacy

While highly effective, the approach has a few limitations. The paper requires the primary network to be fully trained before the secondary one starts, lacking an automated way to determine the optimal transition point. Additionally, the success of the model depends on manually tuning the regression length for the secondary network.

Written in 1992, the networks discussed are tiny by modern standards. However, the core principle of "residual learning"—training a network to predict the errors or residuals of a previous layer—eventually became a foundational concept in deep learning, most notably serving as the cornerstone for modern architectures like ResNets.

Summary updated

BibTeX
@inproceedings{ginzburg1993combined,
  author = {Iris Ginzburg and David Horn},
  title = {{Combined Neural Networks for Time Series Analysis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1993},
}
Figure