Shape-biased CNNs are Not Always Superior in Out-of-Distribution Robustness

Xinkuan Qiu, Meina Kan, Yongbin Zhou, Yanchao Bi, Shiguang Shan

Abstract

Standard Convolutional Neural Networks (CNNs) are known to be "texture-biased," often relying on local patterns rather than global shapes for object recognition. While increasing "shape-bias" has been proposed as a universal solution for out-of-distribution (OOD) robustness, this paper demonstrates that the optimal bias is category-dependent. We introduce Shape-Texture Adaptive Recombination (STAR), a framework that dynamically weights shape and texture features on a per-instance basis, achieving state-of-the-art robustness across multiple benchmarks.

The Problem: Is Shape-Bias a Universal Cure?

Humans are predominantly shape-biased, leading researchers to hypothesize that forcing CNNs to prioritize shape would bridge the gap in OOD robustness—improving performance on blurred, distorted, or artistic images. However, this assumption overlooks two critical factors:

  1. Category Sensitivity: Shape is not always the most discriminative feature. For many animal species, texture (e.g., fur patterns, stripes) is more informative than a generic silhouette.
  2. Scenario Sensitivity: Under certain corruptions like heavy blur, shape information becomes unreliable, making texture a more stable cue.

Proposed Method: STAR

To address these limitations, we propose Shape-Texture Adaptive Recombination (STAR). Rather than enforcing a static bias, STAR learns to adaptively combine features based on the input image.

The architecture consists of:

  • Specialized Heads: A shared backbone branches into three heads: a Shape-biased head ($H_s$) trained on texture-randomized images, a Texture-biased head ($H_t$) trained on shuffled image patches, and a Debiased head ($H_d$) trained with standard augmentations.
  • Recombination Head ($H_r$): This module predicts instance-specific weights ($\alpha, \beta, \gamma$) to fuse the outputs of the specialized heads. For instance, if an image is blurred, $H_r$ shifts weight toward the texture head.

The Animal/Artifact Divide

A key finding of this work is the divergence in feature dependency between natural and man-made objects. As illustrated in Figure 1, texture is often sufficient to identify animals even when poses change, whereas shape is the primary identifier for artifacts.

Figure 1: Comparison of shape and texture information across animal and artifact categories. Texture is highly discriminative for animals (e.g., cats), while shape is more critical for artifacts (e.g., cups vs. furniture).

Our analysis shows that for animals, texture-biased models can outperform shape-biased ones in OOD settings. Conversely, for artifacts, shape remains the dominant and most robust cue.

Experimental Results

We evaluated STAR on a newly curated Category-Balanced ImageNet and several robustness benchmarks (ImageNet-C, ImageNet-R, Sketch, etc.).

  • OOD Robustness: STAR achieved an average robustness of 57.2%, significantly outperforming vanilla ResNet-50 (42.7%) and standard shape-biasing methods like Stylized-ImageNet (50.7%).
  • Adversarial Robustness: STAR showed remarkable resilience to FGSM attacks, scoring 59.3% compared to the baseline's 35.0%.
  • Adaptive Weighting: Qualitative checks confirm that STAR intelligently adjusts its weights—favoring texture for blurred images and shape for pixelated ones.

Conclusion

The quest for robust computer vision should move beyond the "shape-bias is better" dogma. Robustness is not about a single universal bias but about the ability to select the most reliable cues for a given object in a given environment. STAR provides a flexible mechanism to achieve this, paving the way for more adaptable and resilient AI systems.

Summary updated

Abstract

In recent years, Out-of-Distribution (o.o.d) Robustness has garnered increasing attention in Deep Learning, and shape-biased Convolutional Neural Networks (CNNs) are believed to exhibit higher robustness, attributed to the inherent shape-based decision rule of human cognition. In this work, we delve deeper into the intricate relationship between shape/texture information and o.o.d robustness by leveraging a carefully curated "Category-Balanced ImageNet" dataset. We find that shape information is not always superior in distinguishing distinct categories and shape-biased model is not always superior across various o.o.d scenarios. Motivated by these insightful findings, we design a novel method named Shape-Texture Adaptive Recombination (STAR) to achieve higher o.o.d robustness. A category-balanced dataset is firstly used to pretrain a debiased backbone and three specialized heads, each adept at robustly extracting shape, texture, and debiased features. Subsequently, an instance-adaptive recombination head is trained to adaptively adjust the contributions of these distinctive features for each given instance. Through comprehensive experiments, our proposed method achieves state-of-the-art o.o.d robustness across various scenarios such as image corruptions, adversarial attacks, style shifts, and dataset shifts, demonstrating its effectiveness.
BibTeX
@inproceedings{qiu2024shape,
  author = {Xinkuan Qiu and Meina Kan and Yongbin Zhou and Yanchao Bi and Shiguang Shan},
  title = {{Shape-biased CNNs are Not Always Superior in Out-of-Distribution Robustness}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00232},
}
Figure