Predictive Interval Models for Non-parametric Regression

Mohammad Ghasemi Hamed, Mathieu Serrurier, Nicolas DurandarXiv 2014 · DOI · arXiv

TL;DR

This paper introduces a novel method for constructing predictive intervals in non-parametric regression that reliably capture a desired proportion of the conditional distribution of the response variable, even without assuming homoscedasticity. The approach leverages tolerance intervals on prediction errors within local neighborhoods, offering a more robust and precise alternative to existing methods.

What Problem Does This Paper Address?

In regression analysis, we often want to predict not just a single value for the response variable, but a range that is likely to contain it. Traditional methods often provide confidence intervals for the mean of the response, or make strong assumptions about the error distribution (like homoscedasticity). This paper tackles the challenge of creating predictive intervals that guarantee, with a certain confidence, to contain a specified proportion of the conditional distribution of the response variable for any given set of predictors. This is crucial for applications where understanding the full range of possible outcomes is as important as the central tendency, such as in quality control or risk assessment. The authors highlight that existing methods often fall short, especially in non-parametric settings and without homoscedasticity assumptions.

Methods

The core of the proposed method lies in constructing predictive intervals for non-parametric regression by utilizing tolerance intervals on prediction errors. Specifically, they apply this to Local Linear Regression (LLR).

Here's a breakdown of the key methodological contributions:

  • Predictive Interval Models (PIMs): The paper defines PIMs as models that provide predictive intervals for each point in the predictor space, guaranteeing a certain content (proportion of the distribution) on average.
  • Tolerance Intervals for Prediction Errors: Instead of directly modeling the response, the method focuses on the prediction errors. They build tolerance intervals for these errors within local neighborhoods of the query point.
  • Local Linear Regression (LLR): LLR is used as the underlying regression model. The method accounts for potential regression bias and locally varying error variances (heteroscedasticity).
  • LHNPE (Local Homoscedastic Normal Prediction Error) Conditions: This theoretical framework helps in defining the neighborhood for prediction error tolerance intervals, assuming locally constant mean and variance of prediction errors.
  • Hyper-parameter Tuning: A novel optimization algorithm is proposed to tune hyper-parameters (like bandwidths and confidence levels) to find the smallest reliable predictive intervals. This involves a two-phase approach: first, finding the best regression model, and then optimizing interval-specific parameters.
  • PIM Test: A statistical test is introduced to verify if an interval prediction model truly meets the criteria of a predictive interval model.
  • Comparison Measures: New metrics (MIP, MIS, EGSD) are defined to quantitatively compare different interval prediction methods based on reliability, efficiency, and tightness.

Datasets

The authors evaluated their proposed methods on nine benchmark regression datasets:

  • Parkinson1 (from "Parkinsons Telemonitoring")
  • Parkinson2 (from "Parkinsons Telemonitoring")
  • Wine Quality (Red Wine)
  • Concrete Compressive Strength
  • Housing
  • Auto MPG
  • CPU
  • Concrete Slump Test
  • Motorcycle

No new datasets were introduced.

Key Insights and Ideas

  • Focusing on Prediction Error is Key: By building tolerance intervals on prediction errors rather than directly on the response variable, the method can better handle non-homoscedasticity and local variations in the error distribution. This is a crucial departure from many traditional approaches.
  • Local Neighborhoods Matter: The effectiveness of the method hinges on exploiting the local structure of the data and prediction errors. The concept of LHNPE neighborhoods allows for more adaptive interval construction.
  • Reliability and Efficiency Can Be Quantified: The introduction of MIP, MIS, and EGSD metrics provides a structured way to evaluate and compare interval prediction models, moving beyond qualitative assessments.
  • LLR is a Strong Foundation: The paper demonstrates that LLR, when combined with these novel interval construction techniques, can yield highly reliable and efficient predictive intervals.
  • Addressing Bias is Important: The method explicitly accounts for regression bias, which is often ignored in simpler interval prediction techniques.

Key Results

The experimental results across the nine benchmark datasets consistently show that the proposed methods ("Var. K" and "Fixed K") outperform conventional methods like "Loess Conv." and "LS-SVM Conv." in terms of:

  • Reliability: The proposed methods consistently achieve higher Mean Inclusion Percentages (MIP), meaning they more reliably capture the desired proportion of the response variable, especially at higher confidence levels (β = 0.9, 0.95, 0.99).
  • Efficiency: The "Var. K" method, in particular, often yields the smallest EGSD values, indicating more efficient predictive intervals.
  • Tightness: The "Var. K" and "Fixed K" models generally produce tighter reliable bands (smaller MIS ratios), meaning they provide narrower intervals while still maintaining reliability.
  • Robustness: The methods are shown to work well even with datasets exhibiting heteroscedasticity and do not require asymptotic assumptions, making them suitable for smaller datasets.

For instance, in many cases, the proposed methods achieve MIPs significantly higher than the nominal MIP, while conventional methods fail to meet the MIP constraint.

Limitations and Future Work

Limitations:

  • Limited to Local Linear Regression: The proposed methods are currently specialized for Local Linear Regression.
  • Computational Complexity: The methods have a greater computational complexity compared to conventional and quantile regression interval prediction methods.
  • Assumption of Normal Prediction Errors: While the method handles heteroscedasticity, it relies on the assumption of locally homoscedastic normal prediction errors (LHNPE conditions). The paper notes that the methods are not suited when prediction errors are not normally distributed.

Future Work:

  • Generalization to Other Regression Functions: Extending the predictive interval framework to other regression functions, such as Support Vector Machines.
  • One-Sided Interval Prediction: Generalizing the approach to one-sided interval prediction problems.
  • Time Series Models: Applying the methods to interval prediction in time series data.

Summary updated

Abstract

Having a regression model, we are interested in finding two-sided intervals that are guaranteed to contain at least a desired proportion of the conditional distribution of the response variable given a specific combination of predictors. We name such intervals predictive intervals. This work presents a new method to find two-sided predictive intervals for non-parametric least squares regression without the homoscedasticity assumption. Our predictive intervals are built by using tolerance intervals on prediction errors in the query point's neighborhood. We proposed a predictive interval model test and we also used it as a constraint in our hyper-parameter tuning algorithm. This gives an algorithm that finds the smallest reliable predictive intervals for a given dataset. We also introduce a measure for comparing different interval prediction methods yielding intervals having different size and coverage. These experiments show that our methods are more reliable, effective and precise than other interval prediction methods.
BibTeX
@misc{hamed2014predictive,
  author = {Mohammad Ghasemi Hamed and Mathieu Serrurier and Nicolas Durand},
  title = {{Predictive Interval Models for Non-parametric Regression}},
  year = {2014},
  eprint = {1402.5874},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1402.5874},
}
Figure