To what extent should we trust AI models when they extrapolate?

Roozbeh Yousefzadeh, Xuenan Cao

Abstract

Machine learning models are frequently deployed under the assumption that they will primarily perform interpolation—making predictions within the bounds of their training data. However, in practice, models often encounter query points that fall outside the convex hull of the training set, forcing them to extrapolate. This paper presents a geometric auditing framework designed to quantify this "extrapolation gap." By analyzing high-stakes datasets in finance and medicine, we demonstrate that extrapolation is surprisingly common, even in large-scale datasets, and argue that reporting the extent of extrapolation is critical for AI safety and accountability.

Introduction

In the deployment of AI systems for high-stakes decision-making—such as medical diagnosis or credit scoring—trust is often predicated on the model's performance during training and validation. Yet, a fundamental risk remains hidden: the transition from interpolation to extrapolation. Interpolation occurs when a model makes a prediction for a data point situated within the geometric bounds (the convex hull) of its training data. Extrapolation occurs when the model is forced to guess in "uncharted territory" outside these bounds.

Because models are less mathematically constrained when extrapolating, they are significantly more prone to error, bias, and erratic behavior. Despite this, most current AI pipelines provide no indication to the user when a prediction relies on extrapolation. This paper introduces a systematic method for auditing AI models to identify and measure these gaps.

Methodology: Geometric Auditing

The authors propose a model-agnostic geometric analysis to detect extrapolation. The training set $D^{tr}$ is viewed as a finite set of samples forming a convex hull $H^{tr}$. Any query point $x$ that lies outside $H^{tr}$ is classified as an extrapolation.

To quantify the degree of extrapolation, the framework utilizes a constrained optimization approach. Specifically, it solves a constrained least squares problem to project the query point $x$ onto the intersection of the convex hull $H^{tr}$ and the valid feature space $\Omega$. This projection identifies the "closest" point within the training distribution, allowing for a measurement of the distance the model must "reach" to make a prediction.

A key innovation of this method is its ability to handle mixed data types. By binarizing categorical variables and employing homotopy methods, the audit can account for both continuous features (e.g., age, income) and discrete categories (e.g., race, occupation) simultaneously.

Empirical Findings

The framework was tested against three diverse datasets to evaluate how frequently extrapolation occurs in real-world applications.

1. Adult Income Dataset

Using census data to predict annual income, the study found that 52% of testing samples fell outside the training convex hull. Notably, 7% of these samples required extrapolation across both categorical and continuous dimensions simultaneously, representing a significant departure from the training distribution.

2. FICO Dataset

In the context of financial risk, analysis of the FICO dataset revealed that 54% of traders in the test set required extrapolation. The audit identified that features such as "months since oldest trade open" were frequent drivers of extrapolation, suggesting the model was often making decisions for credit profiles it had not previously encountered.

3. Veterans Aging Cohort Study (VACS)

Even with a massive medical dataset containing over 800,000 training samples, 15% of test patients required extrapolation. A case study involving a patient ("Bob") illustrated the clinical danger: the model's prediction for Bob was based on an extrapolation from a training point representing a patient five years younger with significantly different kidney function. Without an audit, a clinician would have no way of knowing the model was operating outside its expertise.

Discussion: The Dimensionality Trap

A recurring theme in the results is that as the number of features (dimensionality) increases, the likelihood of a query point falling outside the hull grows exponentially. In high-dimensional spaces, machine learning almost inevitably becomes an exercise in extrapolation. This "dimensionality trap" underscores the necessity of transparency; if extrapolation is unavoidable, it must at least be visible.

Conclusion and Future Directions

The "extrapolation gap" serves as a vital metric for AI safety. The authors recommend that AI pipelines incorporate "self-auditing" mechanisms that flag predictions requiring excessive extrapolation for human review. While challenges remain—such as the computational cost of solving optimization problems for every query and maintaining data privacy—the shift toward geometric transparency is essential. By knowing when a model is guessing, stakeholders can better manage the risks of AI-driven decision-making in sensitive social and clinical contexts.

Summary updated

BibTeX
@misc{yousefzadeh2022what,
  author = {Roozbeh Yousefzadeh and Xuenan Cao},
  title = {{To what extent should we trust AI models when they extrapolate?}},
  year = {2022},
  eprint = {2201.11260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.11260},
}
Figure