Forecasting Follies: Machine Learning from Human Errors

Li Sun, Yongchen ZhaoJournal of Risk and Financial Management 2025 · DOI

TL;DR

This paper proposes a novel approach to improve inflation forecasts by using machine learning (ML) models to predict and correct human forecasters' errors. The method, akin to an AI-powered grammar checker for forecasts, effectively reduces bias and enhances accuracy, particularly for less skilled forecasters or during periods of high uncertainty.

What Problem Does This Paper Address?

Accurate inflation forecasts are crucial for economic stability and decision-making. While human forecasters are essential, their predictions can be prone to errors due to various factors, including cognitive biases, limited information processing, or periods of structural instability. Existing ML approaches often aim to replace human forecasts entirely or use them as just one input among many. This paper addresses the gap of how ML can augment human forecasts by specifically targeting and correcting their systematic errors, without discarding the valuable insights human forecasters bring.

Methods

The core of the paper's methodology is an "ML-centered forecast adjustment procedure." This involves training ML models (specifically LASSO, elastic net, and ridge regression) to predict the errors of human forecasts. These predicted errors are then used to adjust the original human forecasts. The key novelty lies in this "predict-and-correct" approach, which is designed to be flexible:

  • No Restriction on Human Forecasts: The method works regardless of whether human forecasts are purely judgmental or model-based.
  • Leverages All Information: The ML models utilize predictors, the actual target variable, and the human forecasts themselves.
  • Transparent and Flexible Adjustment: Unlike "black box" models, the ML models are built on standard regression techniques, allowing for interpretation. Human forecasters can choose to partially accept the ML model's suggestions, offering a nuanced adjustment process.

The ML models used are:

  • LASSO (Least Absolute Shrinkage and Selection Operator): Known for variable selection and regularization.
  • Elastic Net: Combines L1 (LASSO) and L2 (ridge) penalties, offering a balance between selection and shrinkage.
  • Ridge Regression: Uses L2 penalty to shrink coefficients, particularly useful when predictors are correlated.

The models are trained to predict forecast errors using macroeconomic and financial variables from the FRED-QD dataset.

Datasets

  • US Survey of Professional Forecasters (SPF) Data: This is the primary dataset for human forecasts, specifically quarterly CPI inflation rate forecasts from 1981Q3 to 2024Q3. The study focuses on frequent participants to mitigate missing data issues.
  • Federal Reserve Economic Data (FRED-QD): A comprehensive dataset of US macroeconomic and financial variables used as predictors for the ML models.
  • Benchmark Forecasts: The authors also generate four sets of benchmark forecasts for comparison:
  • Random Walk: A naive benchmark.
  • AR(1) Model: A simple autoregressive model.
  • Auto ARIMA: A more sophisticated time-series model.
  • SPF Consensus: The median forecast from all SPF participants.

Key Insights and Ideas

  • AI as a "Grammar Checker" for Forecasts: The analogy to spell-checkers is powerful. It suggests ML can help catch and correct "mistakes" in human forecasts without dictating the entire narrative.
  • Human Errors are Predictable: The study demonstrates that systematic errors in human inflation forecasts can be predicted using ML and macroeconomic data.
  • Targeting Large Errors is Key: The adjustment procedures are most effective when human forecasts are already inaccurate or biased, suggesting they are particularly beneficial for less skilled forecasters or during volatile periods.
  • Bias Reduction is a Significant Benefit: Even when forecast accuracy doesn't improve substantially, the ML adjustments can still reduce forecast bias, which is valuable for economic policy.
  • Ridge Regression's Strength in Out-of-Sample: Interestingly, ridge regression, which tends to use more predictors, showed stronger performance in out-of-sample adjustments for large errors, potentially by catching overlooked information.

Key Results

  • In-Sample Improvements: Significant improvements in forecast accuracy (MAE and RMSE) and bias reduction were observed in-sample across various benchmarks.
  • Out-of-Sample Improvements: While more modest, out-of-sample improvements were still evident, especially for less accurate forecasts. For instance, LASSO adjusted forecasts showed lower MAE for 76% and lower RMSE for 95% of cases in-sample.
  • SPF Consensus: The highest quality forecasts (SPF consensus) showed less room for improvement, but even here, some gains were observed.
  • Time-Varying Benefits: The adjustments proved particularly helpful during periods of heightened uncertainty, such as the 2008 recession and the post-pandemic inflation surge.
  • Large Error Mitigation: The procedures were more effective at reducing large forecast errors (e.g., exceeding 1.5 or 2 standard deviations from the mean). For example, out-of-sample, relative RMSEs for errors exceeding 2 standard deviations were reduced to around 0.87-0.91.

Limitations and Future Work

  • Focus on Linear Models: The study primarily uses linear ML models (LASSO, elastic net, ridge). The authors acknowledge that non-linear methods (e.g., random forests) might offer further improvements.
  • Data Availability and Hindsight Bias: While FRED-QD mitigates some hindsight bias, the authors acknowledge the inherent challenge of perfectly replicating forecasters' information sets.
  • Not a "Beat the SPF" Goal: The aim is not to create a fully automated system that outperforms humans, but rather to assist them.
  • Future Work:
    • Investigating optimal utilization of predicted errors in forecast combination and ensemble methods.
  • Extending the procedures to handle uneven and mixed-frequency data.
  • Leveraging the panel structure of the SPF to incorporate information from multiple human forecasters.
  • Exploring non-linear ML techniques.

Summary updated

Abstract

Reliable inflation forecasts are essential for both business operations and macroeconomic policy making. This study explores the potential of using machine learning (ML) techniques to improve the accuracy of human forecasts of inflation. Specifically, we develop and examine ML-centered forecast adjustment procedures where advanced ML techniques are employed to predict and thus mitigate the errors of human forecasts, akin to how an AI-powered spell and grammar checker helps to prevent mistakes in human writing. Our empirical exercises demonstrate the benefits of several popular ML techniques, such as the elastic net, LASSO, and ridge regressions, and provide evidence of their ability to improve both our own benchmark inflation forecasts and those reported by the frequent participants in the US Survey of Professional Forecasters. The forecast adjustment procedures proposed in this paper are conceptually appealing, widely applicable, and empirically effective in reducing forecast bias and improving forecast accuracy.
BibTeX
@article{sun2025forecasting,
  author = {Li Sun and Yongchen Zhao},
  title = {{Forecasting Follies: Machine Learning from Human Errors}},
  journal = {Journal of Risk and Financial Management},
  year = {2025},
  doi = {10.3390/jrfm18020060},
}
Figure