To understand why a complex machine learning model made a specific prediction, we typically ask a counterfactual question: What would change if this feature were different?
Popular feature attribution methods answer this by perturbing the input. For instance, to explain an image classifier, an algorithm might mask a region of pixels with a solid gray box and measure how much the model's confidence drops. For tabular data, it might shuffle a feature's values across the dataset to break its connection to the outcome.
However, this intuitive approach hides a fundamental flaw. When we modify features independently, we break the underlying correlations of the real world. A gray patch in an image, a chemically impossible molecular bond, or a tabular record of a pregnant male are data points that lie far outside the distribution the model was trained on.
When we pass these out-of-distribution (OOD) artifacts into a black-box model, the model's behavior becomes highly unpredictable. The resulting change in prediction might not mean the feature was important; it might simply mean the model is confused by an impossible input.
To solve this, a major research direction focuses on on-manifold perturbations. By constraining feature modifications to stay within the natural data distribution, these techniques produce explanations that are significantly more robust, faithful, and secure.
The Off-Manifold Trap
To understand why standard perturbations fail, consider the concept of the data manifold—the lower-dimensional space where real-world data actually resides. Images of faces, natural text sentences, and valid medical records occupy only a tiny fraction of all possible pixel or token combinations.
When traditional explainers perturb an input, they step off this manifold. This creates three severe problems:
- Artifact Bias: As Chun-Hao Chang et al. demonstrate in Explaining Image Classifiers by Counterfactual Generation, standard masking techniques like blurring or solid color fills introduce unnatural edges and textures. The model reacts to the presence of these strange artifacts rather than the absence of the feature, biasing the importance score.
- Predictive Multiplicity: Many different models can achieve identical accuracy on a dataset while drawing wildly different decision boundaries in the empty spaces outside the data manifold. Klaus Broelemann et al. show in On Counterfactual Explanations under Predictive Multiplicity that "sparse" counterfactuals (which change as few pixels or values as possible without regarding data density) are incredibly fragile. If you update or swap the underlying model, these off-manifold explanations completely break down.
- Adversarial Vulnerability: Because standard explainers like SHAP or LIME evaluate models in empty regions of the feature space, they create a massive security loophole. Attackers can design models that behave maliciously on real data but look perfectly fair during an audit by hiding their biases exclusively in off-manifold regions where real data never occurs (Muhammad Faaiz Taufiq et al. in Manifold Restricted Interventional Shapley Values).
Mechanisms for Staying On the Manifold
How do we alter a feature while respecting the laws of the data distribution? Researchers have developed several distinct mechanisms to keep perturbations firmly on the manifold.
Off-Manifold (Naive) Perturbation:
[Original Input] ───(Replace with Noise/Blur)───> [Impossible OOD State] ──> Erratic Model Response
On-Manifold (Distribution-Aware) Perturbation:
[Original Input] ───(Conditional/Generative)───> [Plausible In-Distribution State] ──> True Feature Importance
1. Generative Counterfactual In-filling
Instead of deleting a feature or replacing it with static noise, we can use conditional generative models to fill in the masked regions with plausible alternatives.
In the image domain, the FIDO framework proposed by Chun-Hao Chang et al. uses a generative model to perform in-painting, ensuring that the modified image looks like a real picture. Similarly, Stefan Kolek et al. introduce a rate-distortion framework in A Rate-Distortion Framework for Explaining Black-box Model Decisions that uses in-painting to replace deleted features with realistic synthetic data, avoiding the misleading model reactions caused by heuristic blur.
This concept spans multiple modalities:
- Text: Esma Balkir et al. use generative models in Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection to create realistic text perturbations, preventing the broken, nonsensical sentences typical of random word masking.
- Chemistry: In Enhancing Chemical Explainability Through Counterfactual Masking, the authors replace influential molecular substructures with chemically valid fragments generated by an AI, rather than zeroing out features and creating physically impossible molecules.
2. Covariance-Aware Sampling and Knockoffs
For tabular data, staying on the manifold means respecting the correlation structure between variables.
One approach is Conditional Permutation. Instead of shuffling a feature blindly, Ahmad Chamma et al. propose Conditional Permutation Importance (CPI) in Statistically Valid Variable Importance Assessment through Conditional Permutations. CPI decomposes a feature into a predictable component (based on other features) and a residual, shuffling only the residual. This preserves the original correlation structure during testing.
Another powerful mechanism is the Model-X Knockoff framework. As explored by Jaime Roquero Gimenez et al. in Knockoffs for the Mass: New Feature Importance Statistics with False Discovery Guarantees., knockoffs are synthetic features that mirror the exact exchangeability and correlation structure of the original data. By using knockoffs as substitutes during feature attribution, Kristin Blesch et al. show in Unfooling SHAP and SAGE: Knockoff Imputation for Shapley Values that you can completely strip away an adversary's ability to hide discriminatory behavior behind OOD data points.
3. Geometric Geodesics
Gradient-based attribution methods like Integrated Gradients calculate importance by interpolating a path between a baseline input and the target input. Standard implementations draw a straight Euclidean line, which inevitably cuts through empty, off-manifold space.
Eslam Zaher et al. solve this in Manifold Integrated Gradients: Riemannian Geometry for Feature Attribution by replacing the straight line with a curved geodesic path that bends along a learned data manifold. By forcing the path to follow the data's true geometry, the gradients are calculated only on realistic inputs, yielding cleaner and more stable saliency maps.
Why It Works: Intuition and Benefits
By shifting from naive perturbations to distribution-aware alternatives, we fundamentally change how the model is evaluated. The benefits manifest in three main areas:
Robustness to Noise and Model Shifts
When a counterfactual explanation is forced to be plausible and adhere to the data manifold, it becomes significantly more stable. Shpresim Sadiku et al. demonstrate in S-CFE: Simple Counterfactual Explanations that manifold-constrained counterfactuals exhibit much higher robustness to input noise. They do not drastically change when the starting point is slightly perturbed because they rely on dense data regions rather than fragile, isolated decision boundaries (André Artelt et al. in Evaluating Robustness of Counterfactual Explanations).
Eliminating Adversarial Blind Spots
Forcing attribution paths or surrogate models to stay on-manifold acts as an inherent denoising filter. In Manifold Integrated Gradients, the authors show that this constraint effectively projects away adversarial noise, preventing human-imperceptible changes from altering the explanation. Because the explainer refuses to look at off-manifold regions, malicious models can no longer hide their true behavior in those empty spaces.
Trustworthy Robustness Audits
When we want to test whether an explanation itself is trustworthy, evaluating it with random noise can be highly misleading—especially for rare minority classes where random perturbations almost always land off-manifold.
As Alex Rodriguez et al. argue in When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances, using on-manifold perturbations provides a much truer assessment of explanation stability. In highly unbalanced datasets, Ilaria Vascotto demonstrates in Assessing Reliability of Explanations in Unbalanced Datasets: a Use-case on the Occurrence of Frost Events that on-manifold neighbors are required to accurately evaluate whether an explanation remains stable across realistic, minority-class scenarios.
Conclusion
The shift toward distribution-aware perturbation represents a maturity milestone for explainable AI. By moving away from the mathematical convenience of independent feature manipulation and embracing the geometric reality of the data manifold, these methods ensure that our explanations reflect how models actually operate in the real world.
Key Foundational Work
Knockoffs for the Mass: New Feature Importance Statistics with False Discovery Guarantees. (2019) — contributed 2 concept elements to this direction.
Interpreting Black Box Models via Hypothesis Testing (2020) — contributed 2 concept elements to this direction.
Explaining Image Classifiers by Counterfactual Generation (2019) — contributed 2 concept elements to this direction.