Learning Risk Preferences in Markov Decision Processes: an Application to the Fourth Down Decision in the National Football League

Nathan Sandholtz, Lucas Y. Wu, Martin L. Puterman, Timothy C. Y. ChanarXiv 2023 · DOI · arXiv

TL;DR

NFL coaches are famously more conservative on fourth down than statistical models suggest they should be. This paper uses inverse optimization to reveal that coaches aren't necessarily "wrong"—they are simply optimizing for "worst-case" scenarios (low quantiles of the outcome distribution) rather than average gains. The study also finds that coaches are becoming more risk-tolerant over time and are significantly "braver" when playing in their opponent's territory.

What Problem Does This Paper Address?

For decades, sports analysts have pointed out a glaring discrepancy: statistical models (like the "4th Down Bot") frequently scream "Go for it!", yet coaches choose to punt or kick a field goal. Traditional analysis labels this behavior as "suboptimal."

However, these models usually assume coaches are risk-neutral, meaning they only care about maximizing the expected (average) value of a play. The authors of this paper argue that coaches might be rational actors with risk-sensitive preferences. The challenge is that we cannot observe a coach's internal risk tolerance directly. This paper develops a mathematical framework to "reverse-engineer" those hidden preferences from actual play-by-play data.

Methods

The authors model a football game as a Markov Decision Process (MDP), where each play transitions the game into a new state (yard line, down, distance). To bridge the gap between observed behavior and theory, they employ several sophisticated techniques:

  • Inverse Optimization (IO): Instead of using a model to prescribe an action, they take the observed actions as "optimal" and solve for the objective function (the risk preference) that makes those actions make sense.
  • Quantile Parameterization: They replace the standard "Expected Value" objective with a quantile function ($Q_\tau$). If a coach optimizes for $\tau = 0.5$, they are risk-neutral. If $\tau < 0.5$, they are risk-averse (focusing on avoiding bad outcomes).
  • Bivariate Monotonic Smoothing: To handle "noisy" data (e.g., cases where a coach rarely goes for it), they use shape-constrained additive models (SCAM) to ensure that the estimated value of a state changes logically with field position and distance to gain.
  • Stratification: Because the MDP state space is massive, the authors simplify the model by stratifying results by win probability, score differential, and time remaining.

Datasets

The study utilizes NFL play-by-play data from the 2014 through 2022 seasons, accessed via the nflfastR R package. This includes detailed contextual information for every play, such as yard line, yards to go, score differential, and time remaining. They also use win probability estimates and "Elo" ratings (team strength) from established models (Baldwin, 2024; FiveThirtyEight) for benchmarking and regression analysis.

Key Insights and Ideas

  • Coaches are "Bottom-Line" Thinkers: The results show that league-wide behavior is consistent with optimizing for the 30th to 40th percentile of outcomes. In plain English: coaches are more worried about the "disaster" of a failed conversion than they are excited about the "average" gain of going for it.
  • The "Half-Field" Psychological Split: Coaches exhibit a "split personality" based on field position. They are significantly more risk-averse in their own half (fear of giving the opponent a short field) than in the opponent's half, even when the statistical math for "going for it" is similar.
  • Desperation Breeds Rationality: In the 4th quarter, when win probability is low, coaches' risk preferences shift dramatically toward $\tau = 0.5$ (risk-neutrality). When the game is on the line and they are losing, they finally start acting like the "optimal" statistical models.
  • The League is Learning: There is a clear upward trend in $\tau$ values from 2014 to 2022. The "analytics revolution" is visible in the data; coaches are slowly shifting from extreme conservatism toward risk-neutrality.

Key Results

  • Quantifying Conservatism: While the "4th Down Bot" typically operates at a risk level near $\tau = 0.55$, the actual league average is closer to $\tau = 0.38$.
  • Performance Correlation: The authors regressed estimated risk preferences against actual "points gained." They found a statistically significant positive association: coaches with higher risk tolerances ($\hat{\tau}$) tend to oversee more productive offenses on fourth down.
  • The "4th Down Bot" isn't Perfect: Interestingly, even the "optimal" win-probability models show some inherent bias in their prescriptions, likely due to selection bias in the historical data they were trained on.

Limitations and Future Work

  • State Space Complexity: To keep the math tractable, the authors excluded variables like timeouts and specific weather conditions from the core MDP, using win probability as a proxy instead.
  • Stationary Policy Assumption: The model assumes that the "league-average" behavior for future plays is stationary, which might not hold as the game evolves.
  • Team-Specific Dynamics: The current model uses league-wide transition probabilities. Future work could incorporate team-specific strengths (e.g., "Does having Patrick Mahomes change a coach's underlying risk quantile?") to provide more granular insights.

Reference: Sandholtz, N., Wu, L., Puterman, M., & Chan, T. C. Y. (2024). Learning Risk Preferences in Markov Decision Processes: an Application to the Fourth Down Decision in the National Football League. arXiv:2309.00756v3.

Summary updated

Abstract

For decades, National Football League (NFL) coaches' observed fourth down decisions have been largely inconsistent with prescriptions based on statistical models. In this paper, we develop a framework to explain this discrepancy using an inverse optimization approach. We model the fourth down decision and the subsequent sequence of plays in a game as a Markov decision process (MDP), the dynamics of which we estimate from NFL play-by-play data from the 2014 through 2022 seasons. We assume that coaches' observed decisions are optimal but that the risk preferences governing their decisions are unknown. This yields an inverse decision problem for which the optimality criterion, or risk measure, of the MDP is the estimand. Using the quantile function to parameterize risk, we estimate which quantile-optimal policy yields the coaches' observed decisions as minimally suboptimal. In general, we find that coaches' fourth-down behavior is consistent with optimizing low quantiles of the next-state value distribution, which corresponds to conservative risk preferences. We also find that coaches exhibit higher risk tolerances when making decisions in the opponent's half of the field as opposed to their own half, and that league average fourth down risk tolerances have increased over time.
BibTeX
@misc{sandholtz2023learning,
  author = {Nathan Sandholtz and Lucas Y. Wu and Martin L. Puterman and Timothy C. Y. Chan},
  title = {{Learning Risk Preferences in Markov Decision Processes: an Application to the Fourth Down Decision in the National Football League}},
  year = {2023},
  eprint = {2309.00756},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.00756},
}
Figure