Dynamic Prediction Length for Time Series with Sequence to Sequence Networks

Mark Harmon, Diego Klabjan

TL;DR

This paper introduces a novel sequence-to-sequence (Seq2Seq) model that dynamically predicts the output length for time series forecasting, unlike traditional models that require a fixed prediction horizon. By developing a new loss function that balances prediction accuracy with output length and a "Confidence Distance" metric, the model can adapt its prediction horizon based on the confidence in its forecasts, leading to improved performance, especially in volatile financial markets.

What Problem Does This Paper Address?

Traditional sequence-to-sequence models, widely used in tasks like language translation, are designed with a fixed input and output sequence length. This works well when the output length is naturally known or can be fixed, such as translating a sentence of a specific length. However, many real-world time series prediction problems, like financial security price forecasting, don't have a predetermined output length. For instance, highly volatile securities are difficult to predict far into the future, while stable ones might allow for longer-term predictions. Existing models struggle with this dynamic nature, requiring manual adjustments or fixed prediction horizons that may not be optimal. This paper addresses the challenge of creating a model that can automatically determine the appropriate prediction length for each time series instance, balancing the trade-off between prediction accuracy and the desired output length.

Methods

The core of the paper's contribution lies in its novel approach to dynamic prediction length within a Seq2Seq framework.

  • Dynamic Output Length: The model allows the prediction length to vary per time series instance during inference.
  • New Loss Function: A custom loss function is introduced that incorporates a confidence measure and a penalty term. This function aims to balance prediction accuracy (measured by Kullback-Leibler divergence) with the output length. The confidence measure, denoted as $G$, is compared against a threshold $\tau$. If $G < \tau$, the prediction at that time step is not considered for the loss, effectively shortening the prediction horizon. A penalty term, controlled by hyperparameter $\lambda$, is added to encourage the model to meet the threshold and generate accurate predictions.
  • Confidence Metrics: Four functions are proposed to measure the confidence ($G$) of a prediction:
  • Maximum: The highest probability from the softmax output.
  • Confidence Distance (CD): The difference between the highest and second-highest probability. This metric is designed to capture uncertainty when the model is "struggling between two different labels."
    • Total Variation (TV): Measures the difference between consecutive probability distributions.
  • Wasserstein/Earth Mover Distance (EMD): Another measure of distance between probability distributions, considered more robust than TV.
  • Masking Functions: Two approaches are used to mask the loss based on the confidence threshold:
  • Indicator Function: A hard mask that completely zeros out the loss if the confidence is below the threshold.
  • Sigmoid Function: A smooth masking function that provides a differentiable approximation, allowing for better gradient flow during training.

The authors found that the Confidence Distance metric combined with the sigmoid masking function yielded the best results.

Datasets

The study utilizes financial time series data for training and evaluation.

  • ETF Dataset: Consists of twenty-two distinct Exchange Traded Funds (ETFs). The data represents the relative price change (returns) over 5-minute intervals.
  • Commodity Dataset: Consists of five distinct commodities, using their price at each time step. For consistency and to align with the ETF dataset, the returns of the commodity dataset were used.

The data was preprocessed using standard normalization techniques (mean and standard deviation from the training set). A walk-forward methodology was employed for evaluation, where the training and testing windows were shifted weekly, with training data being warm-started from the previous window.

Key Insights and Ideas

  • Dynamic Prediction Length is Crucial for Volatility: The paper strongly suggests that for time series with varying volatility (like financial markets), a dynamic prediction length is not just beneficial but essential for achieving optimal performance. Static prediction horizons can be suboptimal, either predicting too far into uncertain future or stopping too early in stable periods.
  • Confidence Distance as a Robust Uncertainty Measure: The "Confidence Distance" metric proved to be particularly effective in capturing model uncertainty. It's more sensitive to situations where the model is indecisive between multiple predictions, which is a key indicator of when to stop predicting.
  • Sigmoid Masking Offers Smoother Training: While the indicator function provides a direct way to mask predictions, the sigmoid function offers a differentiable alternative that leads to better and more reliable prediction F1 scores, even if it's more sensitive to hyperparameters.
  • Hyperparameter Tuning is Key: The choice of hyperparameters $\tau$ (confidence threshold) and $\lambda$ (penalty weight) significantly impacts the trade-off between prediction accuracy and output length. Careful tuning is required to find the optimal balance.

Key Results

  • Improved F1 Scores: The dynamic prediction length model, particularly using the Confidence Distance metric with sigmoid masking, consistently outperformed static Seq2Seq models across both ETF and commodity datasets. For instance, the best dynamic model achieved a significant improvement in F1 score compared to static models.
  • ETF Dataset Performance: The best dynamic model on the ETF dataset achieved an F1 score of 0.503 with a traditional LSTM Seq2Seq structure, outperforming a traditional LSTM structure that only achieved an F1 score of 0.209 for a single prediction.
  • Commodity Dataset Performance: The dynamic model with CD and sigmoid masking performed exceptionally well on the commodity dataset, showing a substantial F1 Gap of 21.2% compared to static models. This highlights the advantage of dynamic prediction in datasets with skewed distributions and high volatility.
  • Hyperparameter Sensitivity: The study found that large values of $\lambda$ tend to lead to inferior models, as the model prioritizes generating more predictions over accuracy. Conversely, $\tau$ plays a crucial role in controlling the output length. The recommended setting for best performance on the ETF dataset was $\tau \in [0.05, 0.5]$ and $\lambda \leq 1.0$.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance of the dynamic model is sensitive to the choice of hyperparameters $\tau$ and $\lambda$. While the paper provides recommended ranges, finding the absolute optimal values might require extensive tuning.
  • Computational Complexity: While the paper mentions choosing traditional LSTM architectures over ACT for faster training, dynamic prediction length inherently adds complexity compared to fixed-length models.
  • Future Work: The paper suggests that the developed framework for dynamic prediction length can be applied to any recurrent neural network architecture. Further research could explore other confidence metrics and masking functions, as well as applying this approach to different types of time series data beyond financial markets. The authors also note that their model only uses price data and does not incorporate external features like news or market announcements, which could be a direction for future exploration.

Summary updated

Abstract

Recurrent neural networks and sequence to sequence models require a predetermined length for prediction output length. Our model addresses this by allowing the network to predict a variable length output in inference. A new loss function with a tailored gradient computation is developed that trades off prediction accuracy and output length. The model utilizes a function to determine whether a particular output at a time should be evaluated or not given a predetermined threshold. We evaluate the model on the problem of predicting the prices of securities. We find that the model makes longer predictions for more stable securities and it naturally balances prediction accuracy and length.
BibTeX
@inproceedings{harmon2020dynamic,
  author = {Mark Harmon and Diego Klabjan},
  title = {{Dynamic Prediction Length for Time Series with Sequence to Sequence Networks}},
  booktitle = {ICAIF},
  year = {2020},
  doi = {10.1145/3383455.3422533},
}
Figure