On the Effects of Data Distortion on Model Analysis and Training

Antonia Marcu, Adam Prügel-Bennett

Abstract

Deep learning models are frequently evaluated using data distortions to probe their internal logic, such as shape vs. texture bias. However, this paper demonstrates that these evaluation methods often introduce "artificial information"—artifacts that models latch onto—leading to biased conclusions. We introduce the Data Interference (DI) index to quantify these effects and propose iOcclusion, a more robust metric for measuring model performance under missing information.

Introduction: The Bias in the Probe

In the quest to understand deep learning "black boxes," researchers often poke and prod models with distorted data. We shuffle image patches to see if a model relies on global shape, or we block out parts of an image to test its "occlusion robustness." But what if the very act of prodding the model changes its behavior?

Common data distortions used to probe vision models, including patch shuffling and various forms of occlusion.

A recent paper, "On the Effects of Artificial Data Modification," by Antonia Marcu and Adam Prügel-Bennett, argues that our most popular tools for interpreting models are fundamentally biased. They demonstrate that distortions introduce "artificial information" that models use as shortcuts, contaminating our understanding of their true inductive biases.

The Problem of Data Interference

To understand if a vision model "sees" like a human, we often use data distortions. For example, if you take an image of a cat, chop it into a 4x4 grid, and shuffle the pieces, a human can’t recognize the "cat shape," but they can still see "cat texture." If a model’s accuracy drops significantly on these shuffled images, we conclude it has a high "shape bias."

The authors argue this logic is a trap. Shuffling isn't a neutral operation; it creates a grid of sharp horizontal and vertical edges that weren't there before. The paper introduces the Data Interference (DI) index to quantify how much a specific distortion causes a model to consistently "hallucinate" a specific class.

The "Truck" Artifact

In CIFAR-10 experiments, the authors found that basic ResNet-18 models consistently misclassified shuffled images as "trucks."

The "Truck" Artifact: Shuffling image patches introduces grid-like edges that the model associates with the 'truck' class, leading to a spike in incorrect predictions for that category.

Why does this happen? The model has learned that strong horizontal and vertical lines are a key feature of trucks. The evaluation method accidentally created "truck-like" features, making the model look less robust than it actually is. This "data interference" means the model isn't necessarily losing shape information; it's being distracted by new, artificial features.

Methodology: Towards Fairer Evaluation

The authors propose two primary tools to mitigate these biases:

  1. Data Interference (DI) Index: This metric measures the consistency of misclassification under distortion. A high DI index indicates that the distortion itself is providing a signal the model recognizes.
  2. iOcclusion (Interplay Occlusion): Standard robustness tests measure accuracy drop when parts of an image are covered. However, models are often unfairly penalized because they are sensitive to the type of mask used (e.g., a black box vs. a noisy patch). iOcclusion looks at the change in the generalization gap as occlusion increases, making the metric invariant to the specific appearance of the occluder.

Key Results and Insights

  • Biased Baselines: Basic models showed high DI indices (up to 2.82 on CIFAR-10), proving they are heavily influenced by shuffling artifacts.
  • Metric Stability: While standard occlusion results varied wildly depending on the mask pattern, iOcclusion provided stable results across different mask types.
  • Augmentation Superiority: Models trained with FMix and CutMix were significantly more resistant to data interference than basic models or those trained with standard MixUp.
  • Direction Over Magnitude: The authors show that the magnitude of a data shift matters less than the direction. If a distortion introduces a bias that helps the model separate classes more cleanly, it can improve representations even if the images look "unnatural."

Conclusion

This research highlights a critical flaw in current model interpretability practices: the tools we use to measure bias are themselves biased. By accounting for Data Interference and using more stable metrics like iOcclusion, researchers can gain a clearer picture of how models actually process information. Future work will focus on developing "clean" probes that do not introduce unintended edge or grid artifacts.

Summary updated

BibTeX
@inproceedings{marcu2022effects,
  author = {Antonia Marcu and Adam Prügel-Bennett},
  title = {{On the Effects of Data Distortion on Model Analysis and Training}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=OUBX2YzSdQ},
}
Figure