Meta-analytics: tools for understanding the statistical properties of sports metrics

Alexander Franks, Alexander D’Amour, Daniel Cervone, Luke Bornn

TL;DR

This paper introduces a "meta-analytic" framework to evaluate the quality of sports statistics themselves. By measuring discrimination (signal vs. noise), stability (consistency over time), and independence (uniqueness of information), the authors provide a toolkit to identify which metrics actually help in decision-making and which are just statistical noise.

What Problem Does This Paper Address?

The world of sports analytics is currently suffering from "metric clutter." Between traditional box scores and modern player-tracking data, coaches and front offices are overwhelmed by hundreds of different statistics. However, there has been very little formal effort to quantify which of these metrics are actually reliable.

For example, if a player shoots 45% from three-point range one season and 35% the next, is that a change in skill, or just random variance? If two different "Advanced Impact" metrics both rank a player highly, are they telling you two different things, or are they just redundant? This paper provides the mathematical rigor to answer these questions, helping analysts distinguish between "signal" (true player ability) and "noise" (luck or context).

Methods

The authors propose three primary "meta-metrics" based on an underlying linear mixed-effects model that decomposes a player's performance into intrinsic skill, context (teammates/system), and chance:

  1. Discrimination: Uses a non-parametric bootstrap approach to estimate the fraction of between-player variance that is due to true ability versus sampling error.
  2. Stability: Measures how much a metric varies season-to-season for the same player after removing the "noise" of chance. It identifies metrics that are sensitive to changes in context or aging versus those that stay steady.
  3. Independence: Employs a Gaussian copula model and Principal Component Analysis (PCA) to determine how much new information a metric provides relative to a set of existing ones.

A key technical contribution is the use of the bootstrap to estimate "intrinsic variation" without assuming a specific parametric distribution (like a Normal or Binomial distribution) for every single metric.

Datasets

The authors evaluated their framework using two primary datasets sourced from Sports Reference:

  • NBA: 70 different metrics for all players from the year 2000 onwards.
  • NHL: 40 different metrics for all players from the year 2000 onwards. Metrics included both traditional box score stats (points, rebounds, goals) and "omnibus" advanced metrics (Win Shares, VORP, Corsi, Fenwick).

Key Insights and Ideas

  • The 3P% Paradox: One of the most striking findings is that raw three-point percentage (3P%) is one of the least stable and least discriminative metrics in basketball. Over 50% of the variation in 3P% between players in a single season is attributed to pure chance.
  • Rate vs. Volume: Metrics based on total playing time (like VORP) are highly discriminative but less stable (because injuries happen), while per-minute rate statistics are more stable but less discriminative.
  • Defensive Stats are Unique: In both the NBA and NHL, defensive "disruption" stats like Steals (STL) and Takeaways (TK) provide some of the most unique, independent information. They don't correlate strongly with other offensive or "overall" metrics, making them vital for a complete player profile.
  • Redundancy in "All-in-One" Metrics: Advanced omnibus metrics (like Win Shares, BPM, and PER) are highly redundant. PCA showed that just 15 components can explain 75% of the variance across 70 NBA metrics, suggesting that many "new" stats are just repackaging the same information.

Key Results

  • Shrinkage Works: The authors demonstrated that using an Empirical Bayes (shrinkage) estimator for 3P% significantly improves both its stability and discrimination compared to the raw percentage.
  • Corsi > Fenwick: In the NHL, the results settled a common community debate: Corsi-based metrics (which include blocked shots) are more reliable and stable than Fenwick-based metrics (which exclude them).
  • Versatility Visualization: Using their PCA-based metrics, the authors could quantitatively show player versatility. For instance, LeBron James ranked highly across three entirely independent principal components, confirming his "do-it-all" reputation through a purely statistical lens.

Limitations and Future Work

The authors are careful to note that their meta-metrics measure internal quality, not relevance. A metric could be perfectly stable, highly discriminative, and completely independent (like a player's zip code), but totally useless for winning games.

Future work suggests adding "Relevance" as a fourth meta-metric—quantifying the predictive relationship between a metric and team wins. Additionally, the authors note that these meta-metrics should be calculated for specific player subgroups (e.g., comparing centers to centers) to avoid results being driven purely by physical differences like height.

Summary updated

Abstract

Abstract In sports, there is a constant effort to improve metrics that assess player ability, but there has been almost no effort to quantify and compare existing metrics. Any individual making a management, coaching, or gambling decision is quickly overwhelmed with hundreds of statistics. We address this problem by proposing a set of “meta-metrics”, which can be used to identify the metrics that provide the most unique and reliable information for decision-makers. Specifically, we develop methods to evaluate metrics based on three criteria: (1) stability: does the metric measure the same thing over time (2) discrimination: does the metric differentiate between players and (3) independence: does the metric provide new information? Our methods are easy to implement and widely applicable so they should be of interest to the broader sports community. We demonstrate our methods in analyses of both NBA and NHL metrics. Our results indicate the most reliable metrics and highlight how they should be used by sports analysts. The meta-metrics also provide useful insights about how to best construct new metrics that provide independent and reliable information about athletes.
BibTeX
@article{franks2016meta,
  author = {Alexander Franks and Alexander D’Amour and Daniel Cervone and Luke Bornn},
  title = {{Meta-analytics: tools for understanding the statistical properties of sports metrics}},
  journal = {Journal of Quantitative Analysis in Sports},
  year = {2016},
  doi = {10.1515/jqas-2016-0098},
}

Loading related content...

Figure