Meta-Analytics: Tools for Understanding the Statistical Properties of Sports Metrics

Alexander Franks, Alexander D’Amour, Daniel Cervone, Luke Bornn

Abstract

In sports, there is a constant effort to improve metrics which assess player ability, but there has been almost no effort to quantify and compare existing metrics. Any individual making a management, coaching, or gambling decision is quickly overwhelmed with hundreds of statistics. We address this problem by proposing a set of "meta-metrics" which can be used to identify the metrics that provide the most unique, reliable, and useful information for decision-makers. Specifically, we develop methods to evaluate metrics based on three criteria: 1) stability: does the metric measure the same thing over time; 2) discrimination: does the metric differentiate between players; and 3) independence: does the metric provide new information? Our methods are easy to implement and widely applicable so they should be of interest to the broader sports community. We demonstrate our methods in analyses of both NBA and NHL metrics. Our results indicate the most reliable metrics and highlight how they should be used by sports analysts. The meta-metrics also provide useful insights about how to best construct new metrics which provide independent and reliable information about athletes.

Introduction

The world of sports analytics is currently suffering from "metric clutter." Between traditional box scores and modern player-tracking data, coaches and front offices are overwhelmed by hundreds of different statistics. However, there has been very little formal effort to quantify which of these metrics are actually reliable. For example, if a player shoots 45% from three-point range one season and 35% the next, is that a change in skill, or just random variance? This paper provides the mathematical rigor to answer these questions, helping analysts distinguish between "signal" (true player ability) and "noise" (luck or context).

Methodology

The authors propose three primary "meta-metrics" based on an underlying linear mixed-effects model that decomposes a player's performance into intrinsic skill, context (teammates/system), and chance. This conceptual framework allows for the evaluation of metrics across seasons and players.

Conceptual framework for decomposing sports metrics into skill, context, and chance components across seasons, players, and metrics.
  1. Discrimination: Uses a non-parametric bootstrap approach to estimate the fraction of between-player variance that is due to true ability versus sampling error.
  2. Stability: Measures how much a metric varies season-to-season for the same player after removing the "noise" of chance. It identifies metrics that are sensitive to changes in context or aging versus those that stay steady.
  3. Independence: Employs a Gaussian copula model and Principal Component Analysis (PCA) to determine how much new information a metric provides relative to a set of existing ones.

A key technical contribution is the use of the bootstrap to estimate "intrinsic variation" without assuming a specific parametric distribution for every single metric.

Datasets and Insights

The framework was evaluated using two primary datasets sourced from Sports Reference: 70 different metrics for NBA players and 40 for NHL players, spanning from the year 2000 onwards. Metrics included traditional box score stats and "omnibus" advanced metrics like Win Shares, VORP, and Corsi.

Key Insights

  • The 3P% Paradox: Raw three-point percentage (3P%) is one of the least stable and least discriminative metrics in basketball. Over 50% of the variation in 3P% between players in a single season is attributed to pure chance.
  • Rate vs. Volume: Metrics based on total playing time (like VORP) are highly discriminative but less stable due to injury risk, while per-minute rate statistics are more stable but less discriminative.
  • Defensive Stats are Unique: In both the NBA and NHL, defensive "disruption" stats like Steals (STL) and Takeaways (TK) provide some of the most unique, independent information, as they do not correlate strongly with offensive metrics.

Results

The analysis yielded several practical applications for sports analysts:

  • Shrinkage Works: Using an Empirical Bayes (shrinkage) estimator for 3P% significantly improves both its stability and discrimination compared to the raw percentage.
  • Corsi > Fenwick: In the NHL, the results settled a common community debate: Corsi-based metrics (which include blocked shots) are more reliable and stable than Fenwick-based metrics (which exclude them).
  • Redundancy in "All-in-One" Metrics: Advanced omnibus metrics (like Win Shares, BPM, and PER) are highly redundant. PCA showed that just 15 components can explain 75% of the variance across 70 NBA metrics, suggesting many "new" stats are just repackaging the same information.

Limitations and Future Work

The authors note that these meta-metrics measure internal quality, not relevance. A metric could be perfectly stable and independent (like a player's height) but not necessarily predictive of winning games. Future work suggests adding "Relevance" as a fourth meta-metric—quantifying the predictive relationship between a metric and team wins. Additionally, meta-metrics should be calculated for specific player subgroups to avoid results being driven purely by physical differences.

Summary updated

Abstract

In sports, there is a constant effort to improve metrics which assess player ability, but there has been almost no effort to quantify and compare existing metrics. Any individual making a management, coaching, or gambling decision is quickly overwhelmed with hundreds of statistics. We address this problem by proposing a set of "meta-metrics" which can be used to identify the metrics that provide the most unique, reliable, and useful information for decision-makers. Specifically, we develop methods to evalute metrics based on three criteria: 1) stability: does the metric measure the same thing over time 2) discrimination: does the metric differentiate between players and 3) independence: does the metric provide new information? Our methods are easy to implement and widely applicable so they should be of interest to the broader sports community. We demonstrate our methods in analyses of both NBA and NHL metrics. Our results indicate the most reliable metrics and highlight how they should be used by sports analysts. The meta-metrics also provide useful insights about how to best construct new metrics which provide independent and reliable information about athletes.
BibTeX
@article{franks2016meta,
  author = {Alexander Franks and Alexander D’Amour and Daniel Cervone and Luke Bornn},
  title = {{Meta-Analytics: Tools for Understanding the Statistical Properties of Sports Metrics}},
  journal = {Journal of Quantitative Analysis in Sports},
  year = {2016},
  doi = {10.1515/jqas-2016-0098},
}
Figure