Missingness Bias in Model Debugging

Saachi Jain, Hadi Salman, Eric Wong, Pengchuan Zhang, Vibhav Vineet, Sai Vemprala, Aleksander Madry

Abstract

Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools. However, in computer vision, pixels cannot simply be removed from an image. One thus tends to resort to heuristics such as blacking out pixels, which may in turn introduce bias into the debugging process. We study such biases and, in particular, show how transformer-based architectures can enable a more natural implementation of missingness, which side-steps these issues and improves the reliability of model debugging in practice.

Introduction

To understand why a computer vision model makes a specific prediction, researchers often employ "removal-based" explanations. For instance, to verify if a model identifies a "dog" based on its head, one might hide the head and observe the change in prediction. While removing words in natural language processing is straightforward, standard Convolutional Neural Networks (CNNs) require a fixed-size grid of pixels. Consequently, practitioners often fill "missing" regions with black pixels, noise, or blur. This paper identifies a fundamental flaw in this approach: missingness bias.

The Missingness Bias

When image regions are replaced with a baseline color like black, CNNs often treat these masks as new features rather than "missing" data. Because these models are typically not trained on images containing sharp black geometric shapes, the masks appear as out-of-distribution artifacts. This leads the model to ignore the actual subject and instead predict classes based on the mask's geometry, such as "crossword puzzle," "jigsaw puzzle," or "maze."

Figure 1: ResNet-50 misclassifying an image as a 'jigsaw puzzle' due to the black mask, while ViT-S remains correct.

This bias renders popular debugging tools like LIME and Integrated Gradients unreliable. For example, changing the masking color from black to grey can completely alter the "important" regions identified by LIME, suggesting that the explanations are often as inconsistent as random noise.

Vision Transformers as a Solution

The researchers propose that Vision Transformers (ViTs) offer a more natural way to handle missingness. Unlike CNNs, which operate on a fixed grid, ViTs process images as a set of tokens. To simulate missing data, tokens can simply be dropped from the input set. This allows the model to make a "best-effort" prediction based solely on the remaining information without being distracted by artificial masks.

Figure 2: Comparison of model predictions when salient features are removed. ViT-S maintains a semantically related prediction ('sea slug') compared to ResNet-50's unrelated 'cliff dwelling'.

Experimental Evaluation

The study evaluates this phenomenon across ImageNet, MS-COCO, and CIFAR-10 using several key metrics:

  • Class Entropy Degradation: As more of an image is hidden, ResNet entropy crashes as it fixates on "mask" classes. In contrast, ViT entropy remains high, indicating it is not being tricked by the removal process.
  • Top-K Ablation: When the most "important" features (according to an explanation) are removed, ViT predictions show a logical and sharp drop in accuracy. On ResNets, the accuracy drop is often similar whether "important" or "random" features are removed, implying the explanations lack meaningful grounding.
  • Consistency: ViT explanations are inherently consistent because they do not rely on an arbitrary baseline color. ResNet explanations show very low agreement (Jaccard similarity) when different masking colors are used.

Conclusion

While retraining CNNs with missingness augmentations can mitigate these biases, it is computationally expensive and involves debugging a surrogate rather than the original model. Vision Transformers, by virtue of their architecture, provide a more robust foundation for reliable model debugging. While not a complete solution for all interpretability challenges, understanding an architecture's relationship with "nothingness" is crucial for developing trustworthy AI.

Code available at: https://github.com/madrylab/missingness

Summary updated

Abstract

Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools. However, in computer vision, pixels cannot simply be removed from an image. One thus tends to resort to heuristics such as blacking out pixels, which may in turn introduce bias into the debugging process. We study such biases and, in particular, show how transformer-based architectures can enable a more natural implementation of missingness, which side-steps these issues and improves the reliability of model debugging in practice. Our code is available at https://github.com/madrylab/missingness
BibTeX
@inproceedings{jain2022missingness,
  author = {Saachi Jain and Hadi Salman and Eric Wong and Pengchuan Zhang and Vibhav Vineet and Sai Vemprala and Aleksander Madry},
  title = {{Missingness Bias in Model Debugging}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=Te5ytkqsnl},
}
Figure