Are All Experts Equally Good? A Study of Analyst Earnings Estimates

Amir Ban, Yishay MansourarXiv 2018 · OpenReview

TL;DR

This paper investigates whether financial analysts have differential expertise in predicting company earnings. While they do possess some individual expertise, the study finds that a significant portion of improved prediction accuracy comes from modeling individual analyst bias, leading to a 20-30% accuracy boost over simple consensus averages.

What Problem Does This Paper Address?

The core question is whether all experts in a given domain are equally good at making predictions. If experts have varying levels of skill (differential expertise), then aggregating their predictions using a weighted average, where more skilled experts get more say, should yield better results than a simple average. The authors use stock market analysts' earnings forecasts as a case study to explore this, aiming to quantify the extent of differential expertise and its impact on prediction accuracy. This is important because if differential expertise exists and can be measured, it opens the door to more accurate forecasting by intelligently combining expert opinions.

Methods

The authors develop a methodology to measure expert differentiation by assessing how much a weighted aggregation of predictions improves upon a simple average. They employ a multi-linear regression model to predict individual analyst forecast errors. This model uses various factors like the time until the earnings announcement, the analyst's prediction frequency, the number of firms they cover, and their past performance with that specific firm. The predicted errors are then used to assign weights to individual analyst forecasts, with smaller predicted errors leading to larger weights. A key innovation is the explicit modeling of individual analyst bias, which is calculated as the average of past prediction errors for a given analyst and firm.

Datasets

The study utilizes historical quarterly earnings forecasts and actual results for large, publicly traded companies. Specifically, they collected data from the Thomson Financial's Institutional Broker Estimate System (I/B/E/S) for companies in the S&P-500 and NASDAQ-100 indices. The data spans from 2004 to 2016, with the improved consensus prediction model being trained and tested on data from 2010-2016.

Key Insights and Ideas

  • Differential Expertise is Modest: While analysts do show some individual expertise, the degree of differentiation is not large enough to be economically significant on its own.
  • Bias is a Major Driver: A substantial portion of the accuracy improvement comes from modeling individual analyst bias, which is found to be surprisingly large and economically important.
  • Individual Bias Matters More Than Firm Bias: Correcting for bias based on individual analysts or analyst-firm combinations yields better results than correcting for general market bias or firm-specific bias alone.
  • Analyst Identity is Key: Replacing analyst identities with their employing institutions (brokerages) leads to decreased accuracy, suggesting that individual bias and expertise are tied to the analyst, not just their employer.
  • Timeliness of Forecasts: Forecasts made closer to the earnings announcement date show more significant differentiation among analysts, indicating that expertise is more pronounced when predictions are more time-sensitive.

Key Results

The study achieves a significant average improvement of 20%-30% in forecast accuracy over standard consensus methods.

  • Modeling individual bias alone improves estimates by 15% (S&P-500) and 22% (NASDAQ-100).
  • Modeling individual expertise (without bias adjustment) yields a more modest improvement of 3% (S&P-500) and 5% (NASDAQ-100).
  • The combined approach, incorporating both bias and expertise, leads to the overall 20-30% improvement.
  • The TREND statistic, which measures the improvement in prediction accuracy, shows a significant positive impact from the proposed methodology.

Limitations and Future Work

The authors acknowledge that their aggregation algorithm is not presented as optimal but as "good enough" to draw qualitative conclusions. They suggest that better algorithms might exist but would need to be significantly superior to change their core findings about modest differentiation. Future work could explore other domains with continuous outcomes and available historical prediction data to see if similar patterns of differential expertise and bias emerge. They are also interested in finding sources of insider data that might be available through agreements.

Summary updated

Abstract

We investigate whether experts possess differential expertise when making predictions. We note that this would make it possible to aggregate multiple predictions into a result that is more accurate than their consensus average, and that the improvement prospects grow with the amount of differentiation. Turning this argument on its head, we show how differentiation can be measured by how much weighted aggregation improves on simple averaging. Taking stock-market analysts as experts in their domain, we do a retrospective study using historical quarterly earnings forecasts and actual results for large publicly traded companies. We use it to shed new light on the Sinha et al. (1997) result, showing that analysts indeed possess individual expertise, but that their differentiation is modest. On the other hand, they have significant individual bias. Together, these enable a 20%-30% accuracy improvement over consensus average.
BibTeX
@misc{ban2018are,
  author = {Amir Ban and Yishay Mansour},
  title = {{Are All Experts Equally Good? A Study of Analyst Earnings Estimates}},
  year = {2018},
  eprint = {1806.06654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.06654},
}
Figure