Clustering-enhanced stock price prediction using deep learning

Man Li, Ye Zhu, Yuxin Shen, Maia Angelova

TL;DR

This paper introduces a novel clustering-enhanced deep learning framework for stock price prediction. By proposing a new similarity measure, Logistic Weighted Dynamic Time Warping (LWDTW), and integrating it with k-medoids clustering, the authors demonstrate improved stock price forecasting accuracy, particularly when combined with LSTM models.

What Problem Does This Paper Address?

Predicting stock prices is a notoriously challenging task due to the inherent complexity, non-stationarity, and chaotic nature of financial time series. While deep learning models like LSTMs have shown promise, their performance can be further enhanced by better understanding and leveraging the relationships between different stocks. Existing studies often use limited datasets or fail to adequately capture the nuanced similarities between stock price movements. This paper aims to address this by developing a more effective method for grouping similar stocks before applying deep learning models for prediction, thereby improving the accuracy and reliability of forecasts.

Methods

The core of this work lies in two main innovations:

  • Logistic Weighted Dynamic Time Warping (LWDTW): This is a novel similarity measure designed to calculate the distance between stock price time series. It extends the Weighted Dynamic Time Warping (WDTW) by incorporating a logistic probability density function to assign weights. This weighting scheme specifically addresses the observed characteristic of financial returns having more extreme observations (fat tails) than a normal distribution. LWDTW aims to underweight extreme returns and emphasize more typical ones, leading to a more robust similarity measure.
  • Clustering-Enhanced Deep Learning Framework: The paper proposes a framework that first applies clustering to group similar stocks. This clustering is performed using the proposed LWDTW, along with standard Euclidean distance and Dynamic Time Warping (DTW) as benchmarks. The k-medoids algorithm is used for clustering. Once stocks are grouped, deep learning models (LSTM, RNN, and GRU) are trained within each cluster. This allows the models to learn parameters that are more specific to the price patterns within that cluster, leading to potentially more accurate predictions. The framework is evaluated in four settings: no clustering, and clustering-enhanced LSTM, GRU, and RNN.

Datasets

The study utilizes daily stock price data for 322 US stocks from January 2005 to November 2017, sourced from Kaggle.com. This dataset covers all 11 S&P sectors and includes stocks that survived the entire period, ensuring data quality and consistency. The data is partitioned into training (10 years), validation (2 years), and testing (11 months) sets.

Key Insights and Ideas

  • Financial Returns Have Fat Tails: The paper highlights that stock returns do not follow a normal distribution but exhibit "fat tails," meaning extreme events are more common than predicted by a normal distribution. This observation is crucial for designing effective similarity measures.
  • LWDTW Outperforms Benchmarks in Clustering: The proposed LWDTW similarity measure consistently yields better clustering results, as evidenced by higher Dunn Index values and lower Davies-Bouldin Index values, compared to Euclidean distance and standard DTW. This suggests LWDTW is more effective at identifying genuinely similar stock price patterns.
  • Clustering Improves Prediction Accuracy: Integrating clustering, especially with LWDTW, significantly improves the forecasting accuracy of deep learning models. This is attributed to the models learning more relevant parameters from stocks with similar price behaviors.
  • LSTM with LWDTW is a Powerful Combination: The combination of LWDTW clustering and the LSTM model demonstrates the best forecasting performance across various evaluation metrics, indicating its superiority for this task.

Key Results

  • Clustering Effectiveness: The LWDTW similarity measure identified optimal cluster numbers (k=4 based on Dunn Index, k=3 based on Davies-Bouldin Index) and effectively grouped stocks with similar price patterns, as visualized in Figure 7 and 9.
  • Forecasting Performance (No Clustering): Among the three deep learning models without clustering, LSTM achieved the best predictive ability, with the lowest MAE (0.0774), MAPE (0.1706), MSE (0.0133), and RMSE (0.0984), and the highest R² (0.9047) on the testing set.
  • Forecasting Performance (With Clustering):
    • MAPE: LWDTW-based clustering consistently yielded the lowest MAPE averages across all models and stock groups (e.g., 0.1278 for all 70 stocks with LSTM).
  • MAE: Similar improvements were observed for MAE, with LWDTW-LSTM achieving the lowest averages (e.g., 0.0536 for all 70 stocks).
  • MSE & RMSE: LWDTW-based clustering also led to lower MSE and RMSE values.
  • R²: The highest R² values were achieved with LWDTW clustering, indicating better overall accuracy (e.g., 0.9517 for all 70 stocks with LSTM).
  • Overall Best Performance: The study concludes that the LWDTW-LSTM combination provides the best forecasting accuracy, with the lowest prediction errors and highest accuracy metrics.

Limitations and Future Work

  • Dataset Scope: While the dataset is substantial (322 stocks, ~1 million observations), it is limited to US stocks and a specific time frame. Future work could explore international markets and longer historical periods.
  • Clustering Algorithm: The paper uses k-medoids clustering. Exploring other advanced clustering algorithms might yield further improvements.
  • Deep Learning Models: While LSTM, RNN, and GRU are powerful, other deep learning architectures (e.g., Transformers) could be investigated for stock price prediction.
  • Feature Engineering: The study primarily uses price data. Incorporating other financial indicators, news sentiment, or macroeconomic data could potentially enhance prediction accuracy.

Summary updated

Abstract

In recent years, artificial intelligence technologies have been successfully applied in time series prediction and analytic tasks. At the same time, a lot of attention has been paid to financial time series prediction, which targets the development of novel deep learning models or optimize the forecasting results. To optimize the accuracy of stock price prediction, in this paper, we propose a clustering-enhanced deep learning framework to predict stock prices with three matured deep learning forecasting models, such as Long Short-Term Memory (LSTM), Recurrent Neural Network (RNN) and Gated Recurrent Unit (GRU). The proposed framework considers the clustering as the forecasting pre-processing, which can improve the quality of the training models. To achieve the effective clustering, we propose a new similarity measure, called Logistic Weighted Dynamic Time Warping (LWDTW), by extending a Weighted Dynamic Time Warping (WDTW) method to capture the relative importance of return observations when calculating distance matrices. Especially, based on the empirical distributions of stock returns, the cost weight function of WDTW is modified with logistic probability density distribution function. In addition, we further implement the clustering-based forecasting framework with the above three deep learning models. Finally, extensive experiments on daily US stock price data sets show that our framework has achieved excellent forecasting performance with overall best results for the combination of Logistic WDTW clustering and LSTM model using 5 different evaluation metrics.
BibTeX
@article{li2023clustering,
  author = {Man Li and Ye Zhu and Yuxin Shen and Maia Angelova},
  title = {{Clustering-enhanced stock price prediction using deep learning}},
  journal = {WWW},
  year = {2023},
  doi = {10.1007/s11280-021-01003-0},
}
Figure