Interpretable Low-Dimensional Regression via Data-Adaptive Smoothing

Wesley Tansey, Jesse Thomason, James G. ScottarXiv 2017 · DOI · arXiv

Abstract

We consider the problem of estimating a regression function in the common situation where the number of features is small, where interpretability of the model is a high priority, and where simple linear or additive models fail to provide adequate performance. To address this problem, we present Maximum Variance Total Variation denoising (MVTV), an approach that is conceptually related both to CART and to the more recent CRISP algorithm, a state-of-the-art alternative method for interpretable nonlinear regression. MVTV divides the feature space into blocks of constant value and fits the value of all blocks jointly via a convex optimization routine. Our method is fully data-adaptive, in that it incorporates highly robust routines for tuning all hyperparameters automatically. We compare our approach against CART and CRISP via both a complexity-accuracy tradeoff metric and a human study, demonstrating that that MVTV is a more powerful and interpretable method.

Introduction

In domains such as urban planning and public policy, regression models must often be interpretable to non-technical stakeholders. While deep learning offers high predictive power, its "black box" nature limits its utility in these contexts. Conversely, traditional linear models frequently fail to capture the complex spatial patterns inherent in low-dimensional data (e.g., 2D or 3D spatial coordinates).

A common objective in spatial analysis is creating "areal data"—partitioning a map into regions of constant value, such as crime rates or demographic metrics. Existing methods like Classification and Regression Trees (CART) often over-smooth, creating overly large, blocky regions that miss local nuances. Newer alternatives like CRISP can under-smooth, resulting in noisy, jagged partitions that are difficult for humans to interpret. This paper introduces Maximum Variance Total Variation (MVTV), a method designed to find the optimal balance between accuracy and human-readability.

Methodology: Maximum Variance Total Variation (MVTV)

MVTV employs a two-step, fully data-adaptive pipeline to generate sharp, interpretable partitions:

  1. Maximum Variance Heuristic for Grid Selection: Rather than using a uniform grid, MVTV adaptively determines the grid granularity ($q$) by selecting the grid that maximizes the sum of variances across all cells. This approach ensures the grid captures the underlying signal effectively, even in regions with sparse data.
  2. Graph-based Total Variation (TV) Denoising: After establishing the grid, MVTV fits values to each cell using a TV penalty. This penalty encourages adjacent cells to share the same value unless the data suggests a significant shift, resulting in "plateaus" of constant values with well-defined boundaries.

Unlike previous methods, MVTV automatically selects all hyperparameters through robust cross-validation routines, removing the need for manual tuning.

Experimental Evaluation

The authors evaluated MVTV using the 2014 Austin Crime Data, binned into a 100x100 grid. The performance was compared against CART and CRISP across statistical and human-centric metrics.

Comparison of regression methods on Austin Crime Data: (a) Raw data, (b) CART, (c) CRISP, and (d) MVTV.

Statistical Performance

MVTV demonstrated superior statistical efficiency, achieving the lowest Akaike Information Criterion (AIC) score of 10,327.58, compared to 11,139.29 for CART and 18,326.33 for CRISP. This indicates that MVTV provides a better trade-off between model complexity and data fit.

Human Interpretability Study

To rigorously measure "interpretability," the authors conducted a study with 207 annotators. Participants were shown data processed by different models and asked to predict the value of a missing central cell. MVTV-processed data led to the lowest human prediction error ($2.75 \times 10^{-2}$). Notably, humans performed better with MVTV summaries than with raw, noisy data, confirming the model's ability to extract meaningful signal.

Discussion and Conclusion

MVTV addresses the "sparsity trap" of previous methods like CRISP by using a graph-based approach that only ties adjacent cells, allowing for flexibility in dense regions and smoothness in sparse ones. Visually, MVTV produces maps that are cleaner than CRISP (avoiding "staircase" artifacts) and more detailed than CART.

While currently optimized for 2D and 3D scenarios, future work includes scaling MVTV to higher-dimensional feature sets and extending the maximum variance heuristic to multivariate outputs. MVTV represents a significant step forward in creating regression models that are both statistically powerful and truly intelligible to human users.

Summary updated

Abstract

We consider the problem of estimating a regression function in the common situation where the number of features is small, where interpretability of the model is a high priority, and where simple linear or additive models fail to provide adequate performance. To address this problem, we present Maximum Variance Total Variation denoising (MVTV), an approach that is conceptually related both to CART and to the more recent CRISP algorithm, a state-of-the-art alternative method for interpretable nonlinear regression. MVTV divides the feature space into blocks of constant value and fits the value of all blocks jointly via a convex optimization routine. Our method is fully data-adaptive, in that it incorporates highly robust routines for tuning all hyperparameters automatically. We compare our approach against CART and CRISP via both a complexity-accuracy tradeoff metric and a human study, demonstrating that that MVTV is a more powerful and interpretable method.
BibTeX
@misc{tansey2017interpretable,
  author = {Wesley Tansey and Jesse Thomason and James G. Scott},
  title = {{Interpretable Low-Dimensional Regression via Data-Adaptive Smoothing}},
  year = {2017},
  eprint = {1708.01947},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.01947},
}
Figure