TL;DR
This paper introduces a novel human-machine ensemble method that dynamically adjusts the number of human forecasters included based on the estimated error of a machine forecaster. This approach aims to improve forecast accuracy and reduce costs by leveraging the strengths of both human intuition and machine learning models, particularly in economic forecasting tasks.
What Problem Does This Paper Address?
Combining forecasts from multiple sources, known as ensemble methods, is a common practice in machine learning and economics. However, traditional ensemble methods often use a fixed combination strategy, regardless of the input data or the specific situation. This inflexibility limits their ability to fully exploit the diverse strengths of different forecasters. Specifically, machine learning models excel when past patterns are present in the data, while human forecasters can incorporate broader contextual information, such as economic policies, and adapt to novel situations. Furthermore, human forecasters incur costs (recruitment, maintenance), which traditional methods don't account for. This paper addresses the challenge of creating a more adaptive and cost-effective ensemble by dynamically integrating human and machine forecasts, considering their respective error characteristics and the specific forecasting context.
Methods
The core of the proposed method is a human-machine ensemble that dynamically determines the optimal number of human forecasters to include. The method starts by estimating the expected error of a machine forecast. If the machine's expected error is low, it relies more on the machine's prediction. Conversely, if the machine's expected error is high, it incorporates forecasts from a group of human forecasters.
The key innovation lies in the dynamic adjustment of the human group size. The authors formulate this as an optimization problem to minimize the expected Mean Squared Error (MSE) of the ensemble. They derive theoretical results that provide a formula for the optimal number of humans ($n^*$) to include, which depends on the variance of the machine's forecast error, the average variance of human forecast errors, and the covariance between machine and human forecast errors. This allows the ensemble to adapt to different inputs by adjusting the balance between machine and human contributions.
Datasets
The study utilizes seven U.S. inflation datasets for evaluation. These datasets cover four different inflation indicators:
- CPI (Consumer Price Index): All items.
- CoreCPI: CPI excluding food and energy.
- PCE (Personal Consumption Expenditures):
- CorePCE: PCE excluding food and energy.
The data is sourced from the Bureau of Labor Statistics and the Bureau of Economic Analysis, with sample periods ranging from January 1957 to October 2016.
Two primary sources of human forecasts were used:
- Livingston Survey: A semi-annual survey (June and December) providing 6-month and 12-month ahead inflation forecasts.
- Survey of Professional Forecasters (SPF): A quarterly survey covering all four inflation indicators.
The paper also mentions the use of a recurrent neural network (RNN) with Long Short-Term Memory (LSTM) units as the machine forecaster and an Autoregressive Moving Average (ARMA(1,1)) model as a benchmark.
Key Insights and Ideas
- Dynamic Adaptation is Crucial: Traditional ensemble methods are too rigid. The ability to dynamically adjust the number of human forecasters based on the machine's estimated error is key to improving accuracy and efficiency.
- Human-Machine Synergy: The method effectively balances the strengths of machines (pattern recognition, quantifiable error) and humans (adaptability to novel situations, diverse information processing).
- Cost-Benefit Analysis: By dynamically selecting the number of humans, the method implicitly considers the cost of human forecasts, aiming to reduce unnecessary expenses when the machine's forecast is reliable.
- Diversity Matters: The performance of the ensemble is strongly linked to the diversity between machine and human forecasts. When machine and human forecasts differ significantly (low cov(εθ, εh)), the ensemble tends to perform better.
Key Results
The proposed human-machine ensemble method demonstrated strong performance on the U.S. inflation datasets:
- It achieved the highest forecast accuracy for four out of seven datasets.
- For two other datasets, it achieved the same accuracy as the best traditional methods.
- In a specific case (CPI-LIV), the method successfully reduced the need for human forecasters for a significant portion of the test set, indicating cost savings.
- The method showed effectiveness in reducing squared errors when the RNN's individual error was large.
Limitations and Future Work
- Assumption of Uniform Human Forecasters: The current method treats all human forecasters as having similar characteristics in terms of average variance and covariance. In reality, human forecasters can have varying levels of expertise and accuracy.
- Parameter Estimation: The method relies on estimating parameters like variance and covariance from training data. While this was done successfully in the experiments, obtaining these values in real-world scenarios might be challenging.
- Future Work: The authors suggest exploring scenarios where human forecasters are not a single type but can be categorized (e.g., experts vs. non-experts, academic vs. industry forecasters). This would allow for a more nuanced and potentially more effective ensemble composition.
Summary updated