Abstract
Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools. However, in computer vision, pixels cannot simply be removed from an image. One thus tends to resort to heuristics such as blacking out pixels, which may in turn introduce bias into the debugging process. We study such biases and, in particular, show how transformer-based architectures can enable a more natural implementation of missingness, which side-steps these issues and improves the reliability of model debugging in practice.
Introduction
To understand why a computer vision model makes a specific prediction, researchers often employ "removal-based" explanations. For instance, to verify if a model identifies a "dog" based on its head, one might hide the head and observe the change in prediction. While removing words in natural language processing is straightforward, standard Convolutional Neural Networks (CNNs) require a fixed-size grid of pixels. Consequently, practitioners often fill "missing" regions with black pixels, noise, or blur. This paper identifies a fundamental flaw in this approach: missingness bias.
The Missingness Bias
When image regions are replaced with a baseline color like black, CNNs often treat these masks as new features rather than "missing" data. Because these models are typically not trained on images containing sharp black geometric shapes, the masks appear as out-of-distribution artifacts. This leads the model to ignore the actual subject and instead predict classes based on the mask's geometry, such as "crossword puzzle," "jigsaw puzzle," or "maze."
This bias renders popular debugging tools like LIME and Integrated Gradients unreliable. For example, changing the masking color from black to grey can completely alter the "important" regions identified by LIME, suggesting that the explanations are often as inconsistent as random noise.
Vision Transformers as a Solution
The researchers propose that Vision Transformers (ViTs) offer a more natural way to handle missingness. Unlike CNNs, which operate on a fixed grid, ViTs process images as a set of tokens. To simulate missing data, tokens can simply be dropped from the input set. This allows the model to make a "best-effort" prediction based solely on the remaining information without being distracted by artificial masks.
Experimental Evaluation
The study evaluates this phenomenon across ImageNet, MS-COCO, and CIFAR-10 using several key metrics:
- Class Entropy Degradation: As more of an image is hidden, ResNet entropy crashes as it fixates on "mask" classes. In contrast, ViT entropy remains high, indicating it is not being tricked by the removal process.
- Top-K Ablation: When the most "important" features (according to an explanation) are removed, ViT predictions show a logical and sharp drop in accuracy. On ResNets, the accuracy drop is often similar whether "important" or "random" features are removed, implying the explanations lack meaningful grounding.
- Consistency: ViT explanations are inherently consistent because they do not rely on an arbitrary baseline color. ResNet explanations show very low agreement (Jaccard similarity) when different masking colors are used.
Conclusion
While retraining CNNs with missingness augmentations can mitigate these biases, it is computationally expensive and involves debugging a surrogate rather than the original model. Vision Transformers, by virtue of their architecture, provide a more robust foundation for reliable model debugging. While not a complete solution for all interpretability challenges, understanding an architecture's relationship with "nothingness" is crucial for developing trustworthy AI.
Code available at: https://github.com/madrylab/missingness
Summary updated