TL;DR
This paper introduces a novel "Loss Discounting Framework" (LDF) for combining and selecting time series forecasting models. LDF offers a flexible and computationally efficient way to dynamically adjust model weights based on their past performance, outperforming existing methods in various forecasting scenarios, particularly those with regime changes.
What Problem Does This Paper Address?
Forecasting in time series often involves a multitude of models, each with its own strengths and weaknesses. Simply picking one model can be suboptimal, while combining them (model averaging) or selecting the best one (model selection) are common strategies. However, existing methods like Bayesian Model Averaging (BMA) can be computationally expensive or struggle with dynamic environments where model performance changes over time (e.g., due to regime shifts). Dynamic Model Averaging (DMA) and Dynamic Model Learning (DML) offer improvements but can be sensitive to hyperparameter choices. This paper aims to bridge this gap by providing a more robust, flexible, and computationally efficient framework for model averaging and selection that can adapt to changing market conditions.
Methods
The core of the proposed method is the Loss Discounting Framework (LDF). It generalizes existing approaches like DMA and DML by introducing a multilevel discounting scheme. Instead of a single layer of model combination, LDF uses multiple layers of "meta-models." Each meta-model is essentially a combination of models from the previous layer, weighted by their past performance using a discount factor. This allows for a more nuanced and adaptive weighting of models over time.
Key features of LDF include:
- Flexible Discounting: LDF allows for time-varying discount factors, enabling models to adapt more quickly to changing data dynamics.
- Multilevel Structure: By creating layers of meta-models, LDF can capture complex interactions and adapt to different levels of model performance.
- General Loss Functions: The framework can utilize any suitable scoring rule (loss function) to evaluate model performance, aligning the evaluation with the specific forecasting goal.
- Model Averaging and Selection: LDF can be configured for both averaging (using softmax) and selection (using argmax) at each layer.
The computational advantage comes from relying on simple additions and multiplications, avoiding computationally intensive methods like MCMC.
Datasets
The paper evaluates LDF on:
- Simulated Data: Generated to mimic time-varying regime switches, allowing for controlled testing of the method's adaptability.
- Foreign Exchange (FX) Forecasts: Using a dataset of G10 currencies from November 1989 to July 2020, incorporating macroeconomic fundamentals.
- US Inflation Forecasts: Quarterly data from 1961/Q1 to 2014/Q4.
The paper compares LDF against several established and state-of-the-art methods, including BMA, DMA, Dynamic Model Learning (DML), Bayesian Predictive Synthesis (BPS), and best N-average.
Key Insights and Ideas
- Adaptability is Key: The LDF's ability to dynamically adjust model weights, especially with time-varying discount factors, is crucial for handling regime shifts and improving forecast accuracy.
- Multilevel Structure Enhances Robustness: Using multiple layers of meta-models makes LDF less sensitive to hyperparameter choices compared to single-layer methods like DMA.
- Tailoring to the Goal: The flexibility to use different loss functions allows LDF to be optimized for specific forecasting objectives, leading to better economic outcomes (e.g., portfolio performance).
- Computational Efficiency Matters: LDF offers a strong performance without the high computational burden of more complex methods, making it practical for large-scale applications.
Key Results
- Superior Performance: LDF consistently outperformed benchmark methods (BMA, DMA, BPS, etc.) in simulation studies and FX forecasting, both in terms of predictive log-scores (MLS) and economic evaluation (Sharpe ratios).
- Robustness: LDF showed greater robustness to hyperparameter choices than DMA.
- FX Forecasting Gains: In FX forecasting, LDF configurations achieved better Sharpe ratios and higher cumulative wealth, demonstrating that improved log-scores translate to tangible economic benefits.
- US Inflation Forecasting: While LDF performed well in other scenarios, it showed limitations in US inflation forecasting when none of the models in the pool could accurately capture certain periods, highlighting the importance of model pool coverage.
Limitations and Future Work
- Model Pool Dependence: LDF, like other model averaging/selection methods, relies on the quality and coverage of the initial model pool. If the true data generating process is not represented, performance can be limited.
- Dynamic Pool: The current framework assumes a fixed pool of forecasters. Future work could explore how to dynamically incorporate new forecasters or handle forecasters dropping out.
- Inter-model Dependencies: The current methodology treats models independently. Investigating how to incorporate inter-dependencies between expert models could be a valuable extension.
- Generalizing Discounting: Exploring more general forms of exponential discounting beyond the current scheme could further enhance the framework.
Summary updated