Dynamic prediction length for time series with sequence to sequence network

Diego Klabjan, Mark Harmon

TL;DR

This paper introduces a novel sequence-to-sequence (Seq2Seq) model that dynamically adjusts its prediction length for time series data. By developing a new loss function that balances prediction accuracy with output length, the model can adapt to varying volatilities in financial securities, outperforming fixed-length prediction models.

What Problem Does This Paper Address?

Traditional sequence-to-sequence models, widely used in tasks like language translation, require a fixed output length. This limitation becomes problematic when dealing with time series data where the optimal prediction horizon can vary significantly. For instance, predicting the price of a highly volatile stock is inherently more challenging further into the future compared to a stable one. Existing methods often rely on manually setting a fixed prediction length or using a sliding window approach, which doesn't dynamically adapt to the inherent characteristics of the data. This paper addresses the challenge of creating a model that can learn to predict a variable number of future time steps for each individual time series, based on its own characteristics and the model's confidence.

Methods

The core innovation of this paper lies in a new loss function designed for Seq2Seq models that allows for dynamic output length. This loss function incorporates a confidence metric, denoted as $G$, which measures the model's certainty about a prediction at a given time step. This confidence is then compared against a threshold, $\tau$. If the confidence falls below this threshold, the model stops generating further predictions for that specific time series.

To achieve this, the authors introduce a penalty term into the standard loss function. This penalty discourages the model from prematurely stopping predictions and encourages it to generate accurate outputs. They explore several functions for $G$, including:

  • Maximum: The highest probability from the softmax output.
  • Confidence Distance (CD): The difference between the highest and second-highest probabilities.
  • Total Variation (TV): The absolute difference between consecutive predictions.
  • Wasserstein/Earth Mover Distance (EMD): A measure of the distance between probability distributions of consecutive predictions.

The paper also investigates two masking functions (indicator and sigmoid) to handle the dynamic nature of the summation in the loss function.

Datasets

The study utilizes financial time series data for training and evaluation. Specifically, they use:

  • ETF Dataset: Consisting of twenty-two distinct ETFs. The data represents the relative price change (returns) over five-minute intervals.
  • Commodity Dataset: Consisting of five distinct commodities. The data represents the price of each security at each time step.

The paper does not introduce new benchmark datasets but uses existing financial time series data.

Key Insights and Ideas

  • Dynamic Prediction Length is Crucial for Volatile Data: The paper strongly suggests that for time series with varying volatility (like financial securities), a dynamic prediction length is not just beneficial but necessary for optimal performance.
  • Confidence Metric as a Stopping Criterion: The idea of using a model-derived confidence metric to dynamically determine when to stop predicting is a key contribution. This moves beyond fixed horizons or manual tuning.
  • Balancing Accuracy and Length: The novel loss function effectively trades off prediction accuracy with the length of the prediction, allowing the model to make longer predictions for more stable securities and shorter ones for more volatile ones.
  • Sigmoid Masking Outperforms Indicator: The sigmoid masking function generally leads to better and more reliable F1 scores compared to the indicator function, even though it's more sensitive to hyperparameters.
  • Confidence Distance is a Robust Metric: Among the confidence metrics explored, Confidence Distance (CD) consistently provides the most accurate predictions, especially when combined with the sigmoid masking function.

Key Results

  • The proposed dynamic prediction length model, particularly using the Confidence Distance metric with sigmoid masking, achieved superior F1 scores compared to static prediction models on both ETF and commodity datasets.
  • For the ETF dataset, the best dynamic model achieved an F1 score that significantly outperformed traditional LSTM models with fixed prediction lengths.
  • The dynamic model showed a notable improvement on the commodity dataset, where price volatility is more pronounced. In some cases, all tested (τ, λ) pairs for the dynamic model outperformed static predictions, highlighting its adaptability.
  • The paper demonstrates that large values of the hyperparameter $\lambda$ can lead to inferior models, as the model prioritizes generating more predictions over accuracy.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance of the dynamic model is sensitive to the choice of hyperparameters $\tau$ and $\lambda$. While the paper provides recommended ranges, optimal values may still require tuning.
  • Feature Engineering: The study primarily uses price data and does not incorporate additional features like news or market announcements, which could potentially improve prediction accuracy.
  • Computational Complexity: While the paper mentions choosing traditional LSTM architectures for faster training, exploring more complex architectures with dynamic prediction capabilities could be a future direction.
  • Generalizability: While tested on financial data, further research could explore the applicability of this dynamic prediction length approach to other time series domains.

Summary updated

Abstract

Recurrent neural networks and sequence to sequence models require a predetermined length for prediction output length. Our model addresses this by allowing the network to predict a variable length output in inference. A new loss function with a tailored gradient computation is developed that trades off prediction accuracy and output length. The model utilizes a function to determine whether a particular output at a time should be evaluated or not given a predetermined threshold. We evaluate the model on the problem of predicting the prices of securities. We find that the model makes longer predictions for more stable securities and it naturally balances prediction accuracy and length.
BibTeX
@inproceedings{klabjan2020dynamic,
  author = {Diego Klabjan and Mark Harmon},
  title = {{Dynamic prediction length for time series with sequence to sequence network}},
  booktitle = {ICAIF},
  year = {2020},
  doi = {10.1145/3383455.3422533},
}
Figure