Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrap

Carlos Mougan, Dan Saattrup NielsenAAAI 2023 · DOI

TL;DR

This paper introduces a novel method for monitoring machine learning models in production by estimating their uncertainty. This approach not only detects when a model's performance is degrading but also uses explainable AI (SHAP values) to pinpoint why it's happening, even without access to true labels. This is crucial for efficient retraining and maintaining model reliability in dynamic environments.

What Problem Does This Paper Address?

Machine learning models, once deployed, can experience performance degradation over time due to changes in the underlying data distribution (known as "data drift" or "distribution shift"). Monitoring these models is essential to ensure they continue to perform reliably. However, a significant challenge arises when the true labels for the incoming data are unavailable, making traditional performance metric monitoring impossible. In such scenarios, detecting distribution shift is possible, but it doesn't directly tell us if the model's performance has actually worsened. This can lead to unnecessary retraining (false positives) or missed opportunities to fix critical issues. The paper aims to address this by providing a method that directly estimates model deterioration and explains its causes, even in unsupervised settings.

Methods

The core of the paper lies in two main contributions:

  • Non-parametric Bootstrap for Uncertainty Estimation: The authors propose a novel method for constructing prediction intervals using a non-parametric bootstrap approach. This method is an extension of existing work (Kumar and Srivastava, 2012) and is designed to provide theoretical guarantees on interval coverage. It aims to improve upon previous state-of-the-art methods, particularly in high-variance scenarios. The key idea is to estimate not just the model's prediction but also the variance of that prediction and the observation noise, combining these to form a robust uncertainty estimate.
  • Explainable AI for Source Identification: To understand why a model is deteriorating, the paper leverages Shapley values. After estimating the model's uncertainty, they train a secondary model (often a gradient boosting model) to predict this uncertainty based on input features. By computing the Shapley values for this secondary model, they can identify which input features are most influential in driving the model's uncertainty, and by extension, its performance degradation. This allows for a more targeted approach to debugging and retraining.

Datasets

The paper evaluates its methods on several real-world regression datasets from the UCI repository. These include:

  • Airfoil Self-Noise
  • Bike Sharing
  • Concrete Strength
  • QSAR Fish Toxicity
  • Forest Fires
  • Parkinsons
  • Power Plant
  • Protein

Additionally, a synthetic dataset was used for some experiments to better control the simulation of distribution shifts.

Key Insights and Ideas

  • Uncertainty as a Proxy for Deterioration: The paper posits that increased model uncertainty is a strong indicator of performance degradation, even when true labels are unavailable. This is a more direct measure than simply detecting distribution shifts.
  • Explainability Beyond Distributional Changes: Classical methods can detect shifts in data distributions, but they don't explain why a model's performance might be affected. SHAP values, when applied to uncertainty estimates, provide this crucial "why," identifying specific features driving the problem.
  • Bootstrap for Robustness: The use of non-parametric bootstrapping for uncertainty estimation is shown to be effective, particularly in handling high-variance models, leading to better prediction interval coverage.
  • Practical Tool for Production: The authors have released an open-source Python package called "doubt" to make these methods accessible to practitioners.

Key Results

  • Uncertainty Method Comparison: The proposed "Doubt" method for prediction intervals showed significantly better coverage than other methods (NASA, MAPIE) in high-variance scenarios (decision tree models), and was on par with the best in other cases.
  • Model Deterioration Monitoring: The "Doubt" monitoring system outperformed classical statistical methods (K-S, PSI) in detecting model deterioration across various model architectures and datasets. While K-S and PSI could detect distribution shifts, they often produced false positives by indicating a shift without a corresponding performance drop.
  • Source Identification: SHAP values, when applied to the uncertainty estimates, successfully highlighted features that were genuinely contributing to model deterioration, distinguishing them from random noise features even when distribution shifts were present. Figure 2 visually demonstrates this, showing SHAP values pinpointing key features while statistical methods only indicate a general shift.

Limitations and Future Work

  • Computational Limitations: The authors acknowledge that their experiments were conducted on relatively small to medium-sized datasets due to computational constraints. They suggest that further work is needed to assess the validity of their results on significantly larger datasets.
  • Scope of Models: The current work focuses on tabular data and non-deep learning models. Extending these methods to deep learning models and other data modalities (e.g., images, text) is a natural direction for future research.
  • Unsupervised Focus: While the paper excels in unsupervised monitoring, exploring how these methods can be integrated into supervised or semi-supervised learning pipelines could be beneficial.

Summary updated

BibTeX
@inproceedings{mougan2023monitoring,
  author = {Carlos Mougan and Dan Saattrup Nielsen},
  title = {{Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrap}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26755},
}
Figure