In the quest for interpretable AI, we often rely on saliency maps—those colorful heatmaps that supposedly show "where the model is looking." But as many researchers have pointed out, these maps can be deceptive, looking plausible to humans while being unfaithful to the model's actual logic.
A new paper, "Certified, Not Explained: Exact Token Erasure in Gated Vision State Space Models," by Sadia Parveen and Dr. Tehseen Zia, offers a more rigorous alternative. Instead of just asking where a model looks, they provide a mathematical guarantee of what a model cannot see. By building on the emerging "Mamba" architecture (Selective State Space Models), they've developed a framework for exact token erasure: a way to prove that specific parts of an image have zero influence on the final prediction.
TL;DR
The authors introduce CG-Mamba, a vision model that provides a "certified support set"—a mathematically guaranteed list of image patches that contribute to a prediction. They prove that by gating the input to a Mamba block's convolution, they can achieve "exact erasure," where changing "closed" tokens results in exactly zero change to the model's output.
What Problem Does This Paper Address?
The paper tackles two major headaches in medical AI:
- Unfaithful Explanations: Traditional saliency methods (like Grad-CAM) often fail "sanity checks," showing the same heatmap even if the model's weights are randomized.
- Concept Leakage: In Concept Bottleneck Models (CBMs)—which predict human-readable traits (like "spiculation" in a lung nodule) before the final diagnosis—hidden information about the label can "leak" through the concept layer, making the model less transparent than it seems.
The authors specifically look at Mamba-based models, which are often praised for their "selection" mechanism. They wanted to know: can we turn this selection into a hard guarantee for interpretability?
Methods
The researchers built CG-Mamba, a CBM for lung nodule malignancy. Its core is a single gated, bidirectional selective state-space (S6) block.
Key technical innovations include:
- The Backbone Gate: A learned mask that "closes" certain image tokens.
- Exact Erasure Conditions: They mathematically identified that to truly erase a token's influence, you must mask the input to the block's causal convolution and ensure closed tokens have zero weight at the readout stage.
- Readout Partitioning: They split the model's hidden channels into dedicated groups for each clinical concept to minimize information leakage between them.
- Path-by-Path Knockout: A rigorous audit procedure that corrupts "closed" tokens with noise to verify that the output remains unchanged.
Datasets
The study uses the LIDC-IDRI dataset, a gold standard for lung cancer screening. It contains CT scans of pulmonary nodules along with radiologist ratings for six concepts: spiculation, margin, sphericity, lobulation, subtlety, and texture.
Key Insights and Ideas
- $\Delta$ is a Red Herring: In Mamba literature, the selective step size ($\Delta$) is usually cited as the selection mechanism. This paper surprisingly finds that gating $\Delta$ is neither necessary nor sufficient for exact erasure. The real gatekeeper is the input to the convolution.
- Certified vs. Explained: The authors argue we should view these gates not as "explanations" (which imply human-like reasoning) but as "certified support sets." It’s a structural guarantee: "I promise these specific pixels did not affect the result."
- Leakage is Deceptive: By decomposing the standard Oracle Impurity Score (OIS), they found that much of what looks like "leakage" is actually just the model being good at predicting a concept from its own features, rather than "cheating" by looking at other concepts.
Key Results
The "knockout" experiments provided striking binary results. When the authors' conditions for erasure were met, the change in the model's output was exactly 0.0000. When they violated even one condition (like removing the convolution mask), the output jumped significantly.
Quantitatively, CG-Mamba holds its own:
- Accuracy: It achieved a pooled AUROC of 0.8425, virtually identical to an ungated Mamba baseline (0.8428).
- Leakage: Partitioning the readout reduced "excess" cross-concept predictability by about 0.024 AUROC points, successfully tightening the bottleneck.
Limitations and Future Work
While the guarantee is mathematically "exact," the authors are careful to note its limits:
- Narrow Scope: The guarantee applies to a fixed gate pattern. If you change the image enough to change the gate itself, the guarantee must be re-evaluated.
- Localization vs. Logic: Label-randomization tests showed that the gates mostly reflect where the nodule is (localization) rather than the specific "logic" of malignancy.
- Scaling to 3D: Initial attempts to move from 2D slices to full 3D volumes underperformed, suggesting that while the theory holds, 3D Mamba models require more tuning and compute to be practical for this task.
This work moves the needle for high-stakes AI by providing a "checkable" form of interpretability—one where a clinician can verify, for any given image, exactly which parts of the scan were ignored.
Summary updated