Abstract
Why can’t neural networks (NN) forecast better? In major super-forecasting competitions, NNs have typically under-performed compared to traditional statistical methods. When they have performed well, the underlying methods have been ensembles of NN and statistical methods. In this study, we evaluate the strengths of a collection of methods for forecasting pandemics such as COVID-19 using NNs, statistical methods, and parameterized mechanistic models. We show that NNs typically under-perform in forecasting COVID-19 active cases, which can be attributed to the lack of training data common for such forecasts. Our test data consists of the top ten countries for active COVID-19 cases early in the pandemic. We found that statistical methods outperform NNs in most cases, though NNs remain powerful pattern finders that may be better utilized in hybrid models.
The Paradox of Pandemic Forecasting
In the world of time-series (TS) forecasting, a persistent paradox exists: despite their massive success in computer vision and natural language processing, Neural Networks often lose to traditional statistical methods in major competitions. This is particularly evident in pandemic modeling. Pandemic data is notoriously difficult: it is often "short" (limited historical data), geographically localized, and subject to noisy reporting. This study investigates why NNs—essentially high-powered pattern finders—fail to provide superior forecasts for active COVID-19 cases compared to models developed decades ago.
Methodology and Data
The researchers compared three distinct modeling philosophies:
- Mechanistic Models: Causal models based on epidemiology, such as SIR and SEIR, using differential equations.
- Statistical Models: Data-driven classical methods including ARIMA, Holt-Winters (HWAAS), TBAT, and Facebook’s Prophet.
- Neural Network Models: State-of-the-art architectures like N-BEATS (residual links) and DeepAR (probabilistic RNNs).
The study utilized a Kaggle dataset of COVID-19 cases from early 2020, focusing on the top ten countries by total confirmed cases. The data was formatted as time series representing the percentage of active cases relative to the total population, using 72 instances for training and a 7-day window for prediction.
Key Results: Statistical Dominance
The quantitative results revealed a clear advantage for "old school" mathematics. Statistical methods (ARIMA, TBAT, HWAAS) outperformed deep learning models in 8 out of 10 countries analyzed.
As shown in the table above, ARIMA was the most robust performer for the US, Italy, and Russia, while TBAT led in four other nations. Neural Networks (N-Beats and DeepAR) only took the lead in France and Brazil, respectively.
Visualizing the Performance Gap
Qualitative analysis of the forecast plots highlighted why NNs struggle. While statistical models generally followed the trajectory of the infection curves, NNs often produced "flat" or overly simplistic forecasts when faced with limited data.
Why NNs Struggle
- The "Data Hunger" Problem: NNs require massive amounts of data to estimate millions of parameters. Pandemic time series are often too short and lack the high-frequency data where NNs typically shine.
- The Random Walk Trap: NNs often perform well on data that mimics a "random walk." In countries like Brazil and France, where the infection curve followed such a pattern, NNs were more competitive.
- Domain Expertise: Statistical and mechanistic models incorporate "prior knowledge" about trend behavior. Purely data-driven NNs start from scratch, making them significantly less sample-efficient in data-sparse environments.
Conclusion and Future Work
The authors suggest that NNs should not be used in isolation for pandemic forecasting. The path forward lies in hybridization—specifically Physics-Informed Neural Networks (PINNs). By embedding epidemiological differential equations (like SIR) directly into the loss function of a network, researchers can force the AI to respect the laws of biology while still benefiting from the pattern-matching power of deep learning.
Summary updated