Reliable and Scalable Variable Importance Estimation via Warm-start and Early Stopping

Zexuan Sun, Garvesh RaskuttiAISTATS 2025

Abstract

Estimating variable importance (VI) in complex models like deep neural networks and gradient-boosted decision trees often involves a trade-off between computational cost and estimation accuracy. While full retraining provides a gold standard, it is computationally prohibitive in high-dimensional settings. Conversely, simple "dropout" methods are fast but often yield biased results. This paper presents a scalable framework utilizing warm-start initialization and early stopping to achieve reliable VI estimates with rigorous theoretical guarantees.

Introduction

Variable Importance (VI) is a cornerstone of model interpretability, quantifying the contribution of individual features to a model's predictive performance. The most accurate way to measure this is through "retraining"—training a new model from scratch without the feature of interest. However, for models with many features, this is computationally infeasible. The common alternative, "dropout," which involves zeroing out features in a pre-trained model, often fails to capture the true underlying relationships, especially when features are correlated.

The Proposed Framework

The authors propose a two-step "Warm-start and Early Stopping" algorithm designed to bridge the gap between the speed of dropout and the accuracy of retraining.

  1. Warm-start Initialization: Instead of random initialization, the reduced model (missing a feature) is initialized with the parameters of the full model. The missing features are replaced with their marginal means, providing a high-quality starting point for optimization.
  2. Early Stopping: The model is updated via a limited number of gradient descent steps. This allows the model to adapt to the missing information without the full cost of retraining or the risk of overfitting to noise in the reduced feature set.

Theoretical Foundation

The core of this work is a unified theoretical framework that treats gradient-based updates as iterative kernel updates. By utilizing Reproducing Kernel Hilbert Space (RKHS) theory and the Neural Tangent Kernel (NTK) for neural networks, the authors prove that the early-stopped model converges to the true reduced function at a rate of $O(N^{-1/2})$. A similar result is derived for GBDTs using a symmetric-tree kernel.

Experimental Evaluation

The method was evaluated on synthetic datasets and the real-world Gas Turbine dataset from the UCI repository.

Handling Feature Correlation

A significant challenge for VI methods is feature correlation. As shown in Figure 1, the proposed early stopping method closely tracks the "True VI" across varying levels of correlation ($\rho$), whereas the standard dropout method becomes increasingly biased as correlation grows.

Comparison of VI estimates across different correlation levels (rho) for Neural Networks and GBDTs.

Speed and Accuracy Trade-offs

In high-dimensional simulations, the early-stopping method achieved errors roughly half those of dropout when compared to the retraining ground truth. Furthermore, as illustrated in Figure 3, the method offers a superior Pareto front for the error-time trade-off, being significantly faster than full retraining while maintaining high accuracy. On the Gas Turbine dataset, the method correctly identified "Turbine Inlet Temperature" as the primary driver of NOx emissions, matching expensive retraining results while dropout produced unreliable rankings.

Error vs. computation time for Neural Networks, highlighting the efficiency of the ES-NN method.

Conclusion

This framework provides a principled approach to variable importance that is both computationally efficient and theoretically sound. By leveraging warm-starts and early stopping, practitioners can obtain reliable interpretability metrics for large-scale neural networks and GBDTs without the prohibitive cost of model retraining. Future work includes refining stopping criteria and extending theoretical guarantees to more general tree structures.

Summary updated

BibTeX
@inproceedings{sun2025reliable,
  author = {Zexuan Sun and Garvesh Raskutti},
  title = {{Reliable and Scalable Variable Importance Estimation via Warm-start and Early Stopping}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
}
Figure