Abstract
Convolutional Neural Networks (CNNs) are known to exhibit a "texture bias," prioritizing local patterns over global shapes. While previous research relied on synthetic cue-conflict datasets, this work investigates texture bias in real-world data. We introduce the Texture Association Value (TAV) and Texture Identification (TID) metrics to quantify how models associate specific textures with object classes. Our findings reveal that "natural adversarial examples" often arise from misaligned textures, where a model's strong association with a background or surface texture overrides the actual object shape.
Introduction
For years, researchers have observed that Convolutional Neural Networks (CNNs) perceive the world differently than humans. While humans primarily rely on shape to identify objects, models often rely on texture. Previous attempts to measure this "texture bias" utilized artificial "cue-conflict" datasets, such as a cat's silhouette filled with elephant skin. However, these synthetic datasets are often hand-selected and contain unnatural texture densities that do not reflect real-world model behavior. This paper bridges the gap by quantifying texture's influence on standard benchmarks like ImageNet and challenging sets like ImageNet-A using real-world imagery.
Methods
The researchers developed a framework consisting of two core components to measure texture influence without relying on synthetic data:
- Texture Association Value (TAV): This metric quantifies the relationship between specific textures and object classes. Using the Prompted Textures Dataset (PTD)—a collection of ~363,000 high-resolution texture images across 56 classes—the authors built a matrix of associations. A high TAV indicates that a model strongly and uniquely associates a specific texture (e.g., "striped") with a specific object (e.g., "zebra").
- Texture Identification (TID): This method allows for labeling the texture of a real-world image without human intervention. By comparing a model's output distribution for a real image to the rows of the TAV matrix, the TID identifies the dominant texture perceived by the model.
To validate the TID method, a human study was conducted involving graduate students. The study found a 61% agreement rate between human evaluators and the automated TID system, significantly outperforming the 25% random baseline.
Key Insights and Results
The Misalignment Hypothesis
The most striking insight explains why "natural adversarial examples" (ImageNet-A) fool models. Over 90% of misclassified samples in ImageNet-A contain textures that are "misaligned" with their true label. For instance, a dragonfly resting on a screen door (grid texture) is frequently misclassified as a "honeycomb" because the model possesses a high TAV for the "grid" $\rightarrow$ "honeycomb" association. In these cases, the texture signal simply overwhelms the shape signal.
Texture Dominance and Performance
Models are not merely biased toward texture; they often treat it as the primary signal for classification. Key findings include:
- Accuracy Gap: Models are up to 66% more accurate on images containing the "dominant" texture for a class compared to those that do not.
- Confidence Gap: There is a 40% difference in model confidence based solely on the presence of expected textures.
- Model Scaling: While larger, modern architectures like ConvNeXt associate with fewer textures per class, they remain heavily influenced by those specific textures.
Discussion and Future Work
This research demonstrates that texture is a primary driver of classification in real-world data, rather than just a secondary bias. The authors acknowledge limitations, such as the TID's dependence on the diversity of the PTD dataset. Future work aims to apply this methodology to other visual features like color and to investigate whether Vision Transformers (ViTs) share the same texture-driven failure modes as CNNs. Ultimately, these metrics could be used to develop training routines that "debias" models by encouraging them to ignore spurious texture-object associations.
Summary updated