The Texture Shortcut Problem
Deep neural networks are notoriously pragmatic. When tasked with classifying an image, they naturally seek the path of least resistance. In standard vision datasets, this path is almost always local texture—high-frequency pixel patterns, color distributions, and background tints—rather than global geometry. As Matthias Minderer et al. observe in Automatic Shortcut Removal for Self-Supervised Representation Learning, self-supervised models frequently exploit these low-level textural shortcuts instead of learning high-level semantic structures.
This "texture bias" creates a profound misalignment between computer vision models and human perception. While a human easily recognizes an object regardless of whether it is drawn as a sketch, painted, or covered in an unusual pattern, standard Convolutional Neural Networks (CNNs) fail catastrophically under such shifts. Robert Geirhos et al. fundamentally demonstrated this vulnerability in ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, showing that standard training regimes prioritize texture over shape to a degree that severely compromises out-of-distribution robustness.
In real-world applications, this brittleness manifests as a failure to generalize across domains. For instance, in medical imaging, models trained on MRI data from one scanner often fail when deployed on another due to subtle variations in scanner-specific noise and contrast textures, even though the underlying anatomy remains identical (Ekaterina Kondrateva et al. in Domain shift in computer vision models for MRI data analysis: an overview). Similarly, evaluating models against common real-world corruptions reveals how easily superficial distortions break standard representations (Dan Hendrycks et al. in Benchmarking Neural Network Robustness to Common Corruptions and Perturbations). To build truly rugged vision systems, we must force neural networks to ignore the superficial "look" of an object and focus on its fundamental geometry.
Breaking Shortcuts via Style and Frequency Manipulation
How do we prevent a network from taking the texture shortcut? The core intuition is to make texture completely uninformative. If an object's texture changes randomly while its shape remains constant, the model cannot rely on texture to minimize its loss function; it is forced to map the label to the invariant geometric structure.
One prominent method is style randomization. By applying arbitrary artistic styles to training images while preserving their underlying content boundaries, the local texture information is decoupled from the semantic class (Robert Geirhos et al.; Philip T. G. Jackson et al. in Style Augmentation: Data Augmentation via Style Randomization). When a model sees a tractor rendered with the texture of a stained-glass window, it can no longer rely on metallic sheen or tire-tread textures; it must look at the global geometric envelope.
This concept can be extended directly into the frequency domain. An image can be mathematically decomposed into an amplitude spectrum (which largely captures local textures and high-frequency noise) and a phase spectrum (which preserves spatial structure and shapes). Jia Li exploited this in Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain by swapping the amplitude spectra between different images during training. Because the texture (amplitude) becomes a chaotic distractor, the network is forced to rely on the stable phase spectrum. Similarly, Hoyong Kwon et al. utilize phase concentration in Phase Concentration and Shortcut Suppression for Weakly Supervised Semantic Segmentation to suppress frequency shortcuts, ensuring that vision models activate on object boundaries rather than background textures that share similar statistical properties.
Stripping Pixels for Pure Geometry
An alternative to corrupting texture is to eliminate it entirely, presenting the network with pure geometric abstractions.
Training models on edge-based representations or "pseudo-sketches" forces an absolute reliance on 2D geometry and spatial arrangements. For example, Nikos Efthymiadis et al. demonstrate in Edge Augmentation for Large-Scale Sketch Recognition without Sketches that large-scale classifiers can be trained without a single human-drawn sketch by converting natural RGB images into randomized Binary Thin Edges (rBTE). This strategy forces the network to learn domain-invariant shape features, which can later be leveraged for single-source domain generalization (Nikos Efthymiadis et al. in Crafting Distribution Shifts for Validation and Training in Single Source Domain Generalization).
Stripping texture also serves as an effective filter for fairness and privacy. Ruichen Yao demonstrates in Improving Fairness in Image Classification via Sketching that reducing images to sketches removes subtle texture patterns that often encode sensitive demographic biases (like skin color or age), preventing the network from using them as classification shortcuts. Taking this abstraction to its logical extreme, Maruthi Narayanan et al. bypass pixels entirely in Shape-Biased Domain Generalization via Shock Graph Embeddings by representing objects as shock graphs—topological skeletons of shape—and classifying them via Graph Neural Networks, achieving complete immunity to color and texture variations.
Architectural Inductive Biases and Distillation
While data manipulation removes texture from the input, we can also design architectures and training objectives that inherently favor shape.
Interestingly, not all architectures process texture the same way. Vision Transformers (ViTs) exhibit a stronger inherent shape bias than standard CNNs due to their dynamic receptive fields and self-attention mechanisms (Muzammal Naseer et al. in Intriguing Properties of Vision Transformers). To enhance this property, Ainkaran Santhirasekaram et al. show in Robust prostate disease classification using transformers with discrete representations that converting continuous pixel data into discrete tokens via vector quantization encourages transformers to prioritize global structural features over high-frequency local textures, significantly improving generalization across different medical imaging hardware.
We can also explicitly distill shape awareness into standard models. Frameworks like InBiaseD align standard visual representations with edge-based features, forcing the network to prioritize global structural geometry over local textures. Alternatively, domain adversarial training can be reframed to treat texture-stripped or stylized images as a "shape-prior" target domain, guiding the model to learn features that remain invariant across texture changes (Dhruva Kashyap et al. in Towards Domain Adversarial Methods to Mitigate Texture Bias).
The Generalization Payoff
Shifting a model's inductive bias from texture to shape is not just about robustness to artistic styles; it unlocks fundamental performance gains across diverse vision tasks.
First, it accelerates learning. Junru Zhao et al. introduced a "pre-pretraining" stage focused on shapes, shading, and reflectance in Perceptual Inductive Bias Is What You Need Before Contrastive Learning, achieving a 2x faster convergence in downstream contrastive learning. Second, it eliminates reliance on spurious correlations, such as background tints or lighting conditions, ensuring that the model tracks invariant object properties (Saachi Jain et al. in Combining Diverse Feature Priors).
Finally, shape-biased representations seamlessly bridge the "reality gap" in synthetic-to-real (sim-to-real) transfer. When training models for 3D body pose estimation (Mona Fathollahi Ghezelghieh in Learning Camera Viewpoint Using CNN to Improve 3D Body Pose Estimation), industrial pose tracking (Niklas Gard et al. in Combining Local and Global Pose Estimation for Precise Tracking of Similar Objects), or monocular depth estimation (Xiaotian Chen et al. in S2R-DepthNet: Learning a Generalizable Depth-Specific Structural Representation), randomizing textures on synthetic 3D models forces the network to learn pure structural orientation and geometry. Because the underlying geometry matches the real world, the model generalizes perfectly to untextured or differently lit real-world objects without ever requiring photorealistic training data.