TL;DR
This paper introduces a novel clustering-enhanced deep learning framework for stock price prediction. By proposing a new similarity measure, Logistic Weighted Dynamic Time Warping (LWDTW), and integrating it with k-medoids clustering, the authors demonstrate improved stock price forecasting accuracy, particularly when combined with LSTM models.
What Problem Does This Paper Address?
Predicting stock prices is a notoriously challenging task due to the inherent complexity, non-stationarity, and chaotic nature of financial time series. While deep learning models like LSTMs have shown promise, their performance can be further enhanced by better understanding and leveraging the relationships between different stocks. Existing studies often use limited datasets or fail to adequately capture the nuanced similarities between stock price movements. This paper aims to address this by developing a more effective method for grouping similar stocks before applying deep learning models for prediction, thereby improving the accuracy and reliability of forecasts.
Methods
The core of this work lies in two main innovations:
- Logistic Weighted Dynamic Time Warping (LWDTW): This is a novel similarity measure designed to calculate the distance between stock price time series. It extends the Weighted Dynamic Time Warping (WDTW) by incorporating a logistic probability density function to assign weights. This weighting scheme specifically addresses the observed characteristic of financial returns having more extreme observations (fat tails) than a normal distribution. LWDTW aims to underweight extreme returns and emphasize more typical ones, leading to a more robust similarity measure.
- Clustering-Enhanced Deep Learning Framework: The paper proposes a framework that first applies clustering to group similar stocks. This clustering is performed using the proposed LWDTW, along with standard Euclidean distance and Dynamic Time Warping (DTW) as benchmarks. The k-medoids algorithm is used for clustering. Once stocks are grouped, deep learning models (LSTM, RNN, and GRU) are trained within each cluster. This allows the models to learn parameters that are more specific to the price patterns within that cluster, leading to potentially more accurate predictions. The framework is evaluated in four settings: no clustering, and clustering-enhanced LSTM, GRU, and RNN.
Datasets
The study utilizes daily stock price data for 322 US stocks from January 2005 to November 2017, sourced from Kaggle.com. This dataset covers all 11 S&P sectors and includes stocks that survived the entire period, ensuring data quality and consistency. The data is partitioned into training (10 years), validation (2 years), and testing (11 months) sets.
Key Insights and Ideas
- Financial Returns Have Fat Tails: The paper highlights that stock returns do not follow a normal distribution but exhibit "fat tails," meaning extreme events are more common than predicted by a normal distribution. This observation is crucial for designing effective similarity measures.
- LWDTW Outperforms Benchmarks in Clustering: The proposed LWDTW similarity measure consistently yields better clustering results, as evidenced by higher Dunn Index values and lower Davies-Bouldin Index values, compared to Euclidean distance and standard DTW. This suggests LWDTW is more effective at identifying genuinely similar stock price patterns.
- Clustering Improves Prediction Accuracy: Integrating clustering, especially with LWDTW, significantly improves the forecasting accuracy of deep learning models. This is attributed to the models learning more relevant parameters from stocks with similar price behaviors.
- LSTM with LWDTW is a Powerful Combination: The combination of LWDTW clustering and the LSTM model demonstrates the best forecasting performance across various evaluation metrics, indicating its superiority for this task.
Key Results
- Clustering Effectiveness: The LWDTW similarity measure identified optimal cluster numbers (k=4 based on Dunn Index, k=3 based on Davies-Bouldin Index) and effectively grouped stocks with similar price patterns, as visualized in Figure 7 and 9.
- Forecasting Performance (No Clustering): Among the three deep learning models without clustering, LSTM achieved the best predictive ability, with the lowest MAE (0.0774), MAPE (0.1706), MSE (0.0133), and RMSE (0.0984), and the highest R² (0.9047) on the testing set.
- Forecasting Performance (With Clustering):
- MAPE: LWDTW-based clustering consistently yielded the lowest MAPE averages across all models and stock groups (e.g., 0.1278 for all 70 stocks with LSTM).
- MAE: Similar improvements were observed for MAE, with LWDTW-LSTM achieving the lowest averages (e.g., 0.0536 for all 70 stocks).
- MSE & RMSE: LWDTW-based clustering also led to lower MSE and RMSE values.
- R²: The highest R² values were achieved with LWDTW clustering, indicating better overall accuracy (e.g., 0.9517 for all 70 stocks with LSTM).
- Overall Best Performance: The study concludes that the LWDTW-LSTM combination provides the best forecasting accuracy, with the lowest prediction errors and highest accuracy metrics.
Limitations and Future Work
- Dataset Scope: While the dataset is substantial (322 stocks, ~1 million observations), it is limited to US stocks and a specific time frame. Future work could explore international markets and longer historical periods.
- Clustering Algorithm: The paper uses k-medoids clustering. Exploring other advanced clustering algorithms might yield further improvements.
- Deep Learning Models: While LSTM, RNN, and GRU are powerful, other deep learning architectures (e.g., Transformers) could be investigated for stock price prediction.
- Feature Engineering: The study primarily uses price data. Incorporating other financial indicators, news sentiment, or macroeconomic data could potentially enhance prediction accuracy.
Summary updated