In the world of time-series forecasting, the prevailing trend is toward increasingly complex "global" models—deep neural networks trained on massive datasets to learn universal patterns. However, a new paper by Valery Manokhin challenges this paradigm, introducing Conformal Expert Pools (CEP). CEP is a training-free, probabilistic forecaster that manages to outperform state-of-the-art learned models while being orders of magnitude faster.
TL;DR
Conformal Expert Pools (CEP) is a training-free probabilistic forecasting method that uses a pool of simple experts and conformal prediction to achieve superior accuracy and better-calibrated prediction intervals than complex neural models like Amazon's DeepNPTS. It requires zero cross-series training, adapts instantly to new data, and runs hundreds of times faster than its competitors.
What Problem Does This Paper Address?
Probabilistic forecasting—providing a range of plausible future values rather than a single number—is critical for decision-making under uncertainty. While deep learning models have made strides here, they are computationally expensive to train and can struggle with "out-of-distribution" series.
A simpler predecessor, Conformal Seasonal Pools (CSP), showed that a training-free approach based on past seasonal values and errors could beat complex models. However, CSP was limited: it only used a single "seasonal-naive" forecast, making it brittle on trending series, short histories, or data with changing volatility. CEP addresses these gaps by evolving from a single forecaster to an adaptive pool of experts.
Methods
CEP is built on two core principles:
- Expert Pools: Instead of relying on one forecast method, CEP maintains a pool of simple, classical point forecasters (the "experts"). These experts are not "trained" in the traditional sense; they have fixed settings and are applied directly to the series' history.
- Empirical Error Pools (Conformal Prediction): CEP doesn't assume a normal distribution for uncertainty. Instead, it builds its predictive distribution by sampling from the actual signed residuals (forecast errors) these experts made on the series' own past.
Crucially, CEP is training-free. No parameters are learned across different series. It adapts to each series individually by selecting the best-performing experts and error distributions from that series' specific history.
Datasets
The authors evaluated CEP using a massive, diverse benchmark:
- CSP Benchmark: 232 datasets containing 24,502 forecast windows, covering domains like energy, transport, finance, and weather.
- fev-bench: 36 additional "fresh" files (1,341 windows) from families like cloud workloads and epidemiology to test generalization outside the primary benchmark.
- Laplace Subset: 2,534 windows from 228 datasets specifically used to benchmark against the "Laplace" online ensemble forecaster.
Key Insights and Ideas
- Simplicity Wins: CEP demonstrates that "global" training isn't a prerequisite for high performance. By focusing on the local history of a series, CEP avoids the overhead and potential pitfalls of cross-series learning.
- Calibration is King: One of the most striking findings is the failure of complex models to provide reliable prediction intervals. While DeepNPTS's 95% intervals only contained the true outcome 57.8% of the time, CEP achieved 92.2% coverage.
- The "Laplace" Rebuttal: The paper provides a rigorous critique of the Laplace forecaster, which had been claimed to "beat (almost) everything." Manokhin shows that on multi-step horizons, Laplace's performance collapses, producing biased point forecasts and pathologically wide intervals (sometimes 15 orders of magnitude wider than the data).
- Efficiency as a Feature: Because it avoids training, CEP is incredibly fast. It can forecast the entire M4 competition (100,000 series) in about 66 seconds, whereas a model like Laplace would take over 50 processor-hours.
Key Results
CEP emerged as the strongest method on the benchmark, achieving the best mean rank across all tested probabilistic scores.
- Accuracy: CEP outperformed CSP on 78% of datasets by CRPS (Continuous Ranked Probability Score) and 84% by MSIS (Mean Scaled Interval Score).
- vs. Deep Learning: CEP achieved lower CRPS than Amazon's DeepNPTS on 88% of datasets.
- Speed: CEP has a median forecast time of just 39 milliseconds. It is roughly 271x faster than Laplace and 819x faster than DeepNPTS.
- Point Forecasts: Even as a point forecaster, CEP is superior, lowering the median MASE (Mean Absolute Scaled Error) from 1.10 (CSP) to 1.01.
Limitations and Future Work
The authors acknowledge that CEP is "deliberately minimal." It does not use covariates (like weather data or holidays) or leverage information from other related series, both of which are standard in many industrial applications.
While CEP provides excellent empirical coverage, it lacks a formal theoretical coverage guarantee for all types of non-stationary temporal dependence. The authors suggest that CEP should now serve as the "mandatory baseline" that any new, complex forecasting model must be required to beat.
Summary updated