TL;DR
This paper presents a rigorous analysis of prominent probabilistic time-series forecasting models, finding that simple baselines like MLPs and Linear Regression often outperform them on point estimation metrics like RMSE and MAPE. The authors emphasize the need for standardized evaluation procedures and large-scale datasets to avoid misleading results in the field.
What Problem Does This Paper Address?
Sales forecasting is crucial for optimizing resources in various industries. Probabilistic time-series models are gaining traction because they can provide uncertainty estimates, aiding in optimal decision-making. However, the field suffers from a lack of standardized, rigorous evaluation. Researchers often use different datasets, training procedures, and baselines, making it difficult to compare models fairly and understand their true performance. This can lead to an overestimation of the capabilities of complex probabilistic models, especially for point estimation tasks which are critical in many real-world applications. The paper aims to address this by systematically evaluating popular probabilistic models against simple baselines under controlled experimental conditions.
Methods
The authors employ two key experimental principles to ensure a fair comparison:
- Large-scale dataset with diverse cross-validation: They introduce a new "EC dataset" comprising 6,032 time series over 1,725 time steps, designed to be more balanced in terms of entropy density than existing benchmarks. Ten cross-validation splits are used to assess robustness.
- Standardized training and hyperparameter selection: All models, including probabilistic ones and baselines, undergo the same training procedures and hyperparameter optimization. This involves using the Adam optimizer with default parameters, a standard weight initialization, and a search for multiple learning rates. For hyperparameter tuning, they use 72 randomized trials or heuristic trials guided by ablation studies.
The paper focuses on three probabilistic models: DeepAR, DeepState, and Prophet, and compares them against four baselines: Moving Average (MA), Linear Regression (LR), Multi-layer Perceptron (MLP), and Seasonal ARIMA (SARIMA).
Datasets
- EC Dataset: A newly constructed large-scale real-world sales dataset from an e-commerce website. It consists of 6,032 time series, each representing daily sales for a shop over 1,725 time steps. Shops with sparse sales records (many zeros) were filtered out to ensure predictability and focus on time-series learning.
- Publicly available datasets: Mentioned for comparison in terms of forecastability and spectral entropy density (e.g., traffic, electricity, Wikipedia).
Key Insights and Ideas
- The "Probabilistic" Premium Might Be Overstated for Point Estimation: The most striking finding is that simple, non-probabilistic models like MLPs and Linear Regression can outperform complex probabilistic models on standard point estimation metrics (RMSE, MAPE). This challenges the common assumption that probabilistic models inherently offer superior forecasting accuracy.
- Standardized Evaluation is Crucial: The paper strongly advocates for rigorous, standardized experimental setups. Without them, it's easy to unfairly favor new models and misinterpret progress in the field. The authors' controlled experiments highlight how factors like dataset choice and hyperparameter tuning can significantly influence results.
- Feature Engineering vs. Architecture: The results suggest that for sales forecasting, well-tuned simple models with appropriate feature engineering (even just lagged values) can be more effective than sophisticated architectures that might struggle to capture relevant patterns without extensive tuning or specific data characteristics.
- DeepAR's Sensitivity to Lagged Values: DeepAR's performance was found to be unstable depending on which lagged values were used, suggesting that autoregressive architectures might not effectively convey information from distant past observations over time.
- DeepState's Challenges with Flexibility: The fully-learnable SSM architecture in DeepState, while theoretically more flexible, led to significantly worse performance and higher variance, possibly due to non-identifiability problems and overfitting.
Key Results
- MLP and LR Outperform Probabilistic Models: On the EC dataset, MLP achieved the best RMSE (72.23 ± 12.60) and MAPE (0.267 ± 0.015), outperforming all probabilistic models. Linear Regression also performed competitively.
- DeepAR's NQL Performance: DeepAR showed the best Mean NQL (Normalized Quantile Loss) among the probabilistic models (0.132 ± 0.026), indicating it's better at capturing the predictive distribution's tails than DeepState or Prophet. However, this did not translate to better point estimation.
- Scaling is Critical for DeepAR/DeepState: The absence of scaling significantly degraded performance for both DeepAR and DeepState, with DeepState even encountering errors in Kalman filtering.
- Prophet's Limitations: Prophet struggled with inconsistent seasonality patterns in the EC dataset and performed worse than baselines.
Limitations and Future Work
- Limitations:
- The study focuses on sales forecasting, and results might differ for other domains.
- While the EC dataset is large, it's still a specific type of sales data.
- The paper acknowledges that point estimation (RMSE, MAPE) isn't the only way to evaluate probabilistic models; capturing the distribution is also important.
- Future Work:
- Further research into selecting appropriate probability distributions for time-series models, as current work often defaults to Gaussian. This could improve both point estimation and distributional accuracy.
- Developing probabilistic models that can increase stability and resolve issues like non-identifiability, especially for SSM-based models.
- Investigating how to better convey information from distant past observations in autoregressive models.
Summary updated