A unified framework for evaluating the robustness of machine-learning interpretability for prospect risking

Prithwijit Chowdhury, Ahmad Mustafa, Mohit Prabhushankar, Ghassan AlRegib

Abstract

In geophysics, hydrocarbon prospect risking involves assessing the risks associated with hydrocarbon exploration by integrating data from various sources. Machine learning-based classifiers trained on tabular data have been recently used to make faster decisions on these prospects. The lack of transparency in the decision-making processes of such models has led to the emergence of explainable AI (XAI). LIME and SHAP are two such examples of these XAI methods which try to generate explanations of a particular decision by ranking the input features in terms of importance. However, explanations of the same scenario generated by these two different explanation strategies have shown to disagree or be different, particularly for complex data. This is because the definitions of "importance" and "relevance" differ for different explanation strategies. Thus, grounding these ranked features using theoretically backed causal ideas of necessity and sufficiency can prove to be a more reliable and robust way to improve the trustworthiness of the concerned explanation strategies. We propose a unified framework to generate counterfactuals as well as quantify necessity and sufficiency and use these to perform a robustness evaluation of the explanations provided by LIME and SHAP on high dimensional structured prospect risking data. This robustness test gives us deeper insights into the models' capabilities to handle erroneous data and which XAI module works best in pair with which model for our dataset for hydrocarbon indication.

Introduction

Prospect risking is a high-stakes process in geophysics, determining the likelihood of hydrocarbon presence at a site. Traditionally, experts rely on Direct Hydrocarbon Indicators (DHIs), such as seismic patterns like "gas chimneys" or "bright spots." While machine learning (ML) models offer automation, their "black box" nature necessitates the use of Explainable AI (XAI) tools like LIME and SHAP. However, these tools often produce conflicting feature rankings for the same data point, leading to a "disagreement problem" that undermines trust. This research addresses this by introducing a framework for causal grounding of AI explanations using counterfactual reasoning.

Methods

The proposed framework moves beyond simple feature ranking by employing Forward Counterfactual Generation. This model-agnostic, post-hoc approach calculates two primary metrics to "fact-check" AI explanations:

  1. Necessity Score ($\alpha$): Measures if changing a specific feature alters the model's prediction. If yes, that feature was necessary for that specific outcome.
  2. Sufficiency Score ($\beta$): Measures if keeping a feature constant while perturbing others maintains the prediction. If yes, that feature was sufficient to trigger the outcome.

By perturbing observed data points, the framework creates a "Global Importance Score" that anchors XAI outputs in geophysical reality, avoiding the computational overhead of existing methods like DiCE.

Experimental Setup

The study utilized a proprietary DHI dataset of over 200 drilled prospects. The data includes 35 total attributes, including 7 "high-level" features such as Anomaly Class, DHI Index, and data quality scores. The target is a binary classification of prospect success (hydrocarbon present) or failure. Models tested include Logistic Regression, Gaussian Naive Bayes, Random Forest, and a Voting Classifier, using a 70/30 train-test split.

Results and Discussion

The evaluated models achieved validation accuracies between approximately 71% and 75%. Notably, performance remained relatively stable even when restricted to the seven high-level features compared to the full feature set.

Table 1: Training and Validation Accuracy across different models and feature sets.

A key finding is that importance rankings do not always equate to causality. In several instances, features ranked second by XAI tools exhibited higher necessity scores than those ranked first. The robustness audit revealed that LIME paired with Random Forest was particularly effective. Furthermore, SHAP's rankings showed strong alignment with feature sufficiency, making it a reliable choice for this geophysical application.

Table 2: Top k-th occurrences of high-level features in SHAP explanations for the Random Forest model.

Conclusion

This framework provides a robust method for evaluating AI explanations in the oil and gas industry. While current methods assume feature independence—a limitation given the correlated nature of seismic attributes—future work will incorporate complex causal models to account for confounding variables. This approach not only validates ML models but also assists domain experts in refining their subjective data grading for more reliable prospect risking.

Summary updated

Abstract

In geophysics, hydrocarbon prospect risking involves assessing the risks associated with hydrocarbon exploration by integrating data from various sources. Machine learning-based classifiers trained on tabular data have been recently used to make faster decisions on these prospects. The lack of transparency in the decision-making processes of such models has led to the emergence of explainable AI (XAI). LIME and SHAP are two such examples of these XAI methods which try to generate explanations of a particular decision by ranking the input features in terms of importance. However, explanations of the same scenario generated by these two different explanation strategies have shown to disagree or be different, particularly for complex data. This is because the definitions of "importance" and "relevance" differ for different explanation strategies. Thus, grounding these ranked features using theoretically backed causal ideas of necessity and sufficiency can prove to be a more reliable and robust way to improve the trustworthiness of the concerned explanation strategies.We propose a unified framework to generate counterfactuals as well as quantify necessity and sufficiency and use these to perform a robustness evaluation of the explanations provided by LIME and SHAP on high dimensional structured prospect risking data. This robustness test gives us deeper insights into the models capabilities to handle erronous data and which XAI module works best in pair with which model for our dataset for hydorcarbon indication.
BibTeX
@misc{chowdhury2026unified,
  author = {Prithwijit Chowdhury and Ahmad Mustafa and Mohit Prabhushankar and Ghassan AlRegib},
  title = {{A unified framework for evaluating the robustness of machine-learning interpretability for prospect risking}},
  year = {2026},
  eprint = {2602.14430},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.14430},
}
Figure