Abstract
Estimating variable importance (VI) in complex models like deep neural networks and gradient-boosted decision trees often involves a trade-off between computational cost and estimation accuracy. While full retraining provides a gold standard, it is computationally prohibitive in high-dimensional settings. Conversely, simple "dropout" methods are fast but often yield biased results. This paper presents a scalable framework utilizing warm-start initialization and early stopping to achieve reliable VI estimates with rigorous theoretical guarantees.
Introduction
Variable Importance (VI) is a cornerstone of model interpretability, quantifying the contribution of individual features to a model's predictive performance. The most accurate way to measure this is through "retraining"—training a new model from scratch without the feature of interest. However, for models with many features, this is computationally infeasible. The common alternative, "dropout," which involves zeroing out features in a pre-trained model, often fails to capture the true underlying relationships, especially when features are correlated.
The Proposed Framework
The authors propose a two-step "Warm-start and Early Stopping" algorithm designed to bridge the gap between the speed of dropout and the accuracy of retraining.
- Warm-start Initialization: Instead of random initialization, the reduced model (missing a feature) is initialized with the parameters of the full model. The missing features are replaced with their marginal means, providing a high-quality starting point for optimization.
- Early Stopping: The model is updated via a limited number of gradient descent steps. This allows the model to adapt to the missing information without the full cost of retraining or the risk of overfitting to noise in the reduced feature set.
Theoretical Foundation
The core of this work is a unified theoretical framework that treats gradient-based updates as iterative kernel updates. By utilizing Reproducing Kernel Hilbert Space (RKHS) theory and the Neural Tangent Kernel (NTK) for neural networks, the authors prove that the early-stopped model converges to the true reduced function at a rate of $O(N^{-1/2})$. A similar result is derived for GBDTs using a symmetric-tree kernel.
Experimental Evaluation
The method was evaluated on synthetic datasets and the real-world Gas Turbine dataset from the UCI repository.
Handling Feature Correlation
A significant challenge for VI methods is feature correlation. As shown in Figure 1, the proposed early stopping method closely tracks the "True VI" across varying levels of correlation ($\rho$), whereas the standard dropout method becomes increasingly biased as correlation grows.
Speed and Accuracy Trade-offs
In high-dimensional simulations, the early-stopping method achieved errors roughly half those of dropout when compared to the retraining ground truth. Furthermore, as illustrated in Figure 3, the method offers a superior Pareto front for the error-time trade-off, being significantly faster than full retraining while maintaining high accuracy. On the Gas Turbine dataset, the method correctly identified "Turbine Inlet Temperature" as the primary driver of NOx emissions, matching expensive retraining results while dropout produced unreliable rankings.
Conclusion
This framework provides a principled approach to variable importance that is both computationally efficient and theoretically sound. By leveraging warm-starts and early stopping, practitioners can obtain reliable interpretability metrics for large-scale neural networks and GBDTs without the prohibitive cost of model retraining. Future work includes refining stopping criteria and extending theoretical guarantees to more general tree structures.
Summary updated