Modern computer vision models are remarkably proficient at understanding the world through a photorealistic lens. Trained on millions of natural images, deep convolutional neural networks (CNNs) and Vision Transformers (ViTs) easily recognize objects, detect boundaries, and estimate poses. However, this proficiency relies on a fragile shortcut: local textures.
When these models are evaluated on non-photorealistic domains—such as sketches, cartoons, oil paintings, or caricatures—their performance drops precipitously. This phenomenon is known as the non-photorealistic domain gap or the cross-depiction problem. As Nicholas Westlake demonstrates in Detecting People in Artwork with CNNs, models optimized for photographic data overfit to the specific stylistic characteristics of their training data, failing to generalize when those superficial features are altered.
The Shape-Texture Conflict
To understand why models fail, we must look at how they learn. Human vision is highly robust to abstraction; we can recognize a dog whether it is depicted in a high-resolution photograph, a minimalist line drawing, or an exaggerated caricature. We achieve this by prioritizing global shape and structural topology over local surface patterns.
In contrast, standard deep learning models are notoriously "lazy." They exploit local statistical regularities like pixel gradients, color distributions, and fine textures because they are easier to optimize than complex global geometries. In an artistic rendering or a hand-drawn sketch, these photographic textures are stripped away or replaced by brushstrokes, cross-hatching, or solid fills. Because the model's feature extractors look for photographic artifacts that do not persist across artistic mediums, the semantic representations collapse. As David Kadish et al. observe in Improving Object Detection in Art Images Using Only Style Transfer, standard models rely heavily on texture cues for identification, whereas art frequently prioritizes shape and form, directly subverting this reliance.
Breaking Texture Bias via Structural Priors
If the root cause of the domain gap is an over-reliance on texture, the most direct solution is to systematically destroy texture during training, forcing the network to learn shape-based representations instead.
A powerful mechanism for achieving this is introduced by Zhenlin Xu et al. in Robust and Generalizable Visual Representation Learning via Random Convolutions. The authors propose RandConv, a data augmentation technique that passes images through random, non-learned convolutional filters during training. These random filters scramble local textures, colors, and contrast while preserving the underlying global structures and boundaries. By forcing the network to classify images despite these randomized textures, the model is effectively prevented from using texture shortcuts, leading to a massive boost in generalization on abstract domains like sketch recognition.
Another approach is to explicitly penalize the model for relying on texture. Dhruva Kashyap et al. explore this in Towards Domain Adversarial Methods to Mitigate Texture Bias. The authors utilize Domain Adversarial Training (DAT) to treat edge maps and highly stylized images as a target domain. By framing the training objective as a mini-max game where the feature extractor must fool a domain classifier, the network is forced to discard domain-specific textural properties and instead rely on invariant structural features—acting as an explicit "shape-prior."
Learning Style-Invariant Representations
Rather than modifying the input space to eliminate texture, researchers also target the latent space, designing architectures that project diverse visual styles into a single, unified representation.
In Fuse and Attend: Generalized Embedding Learning for Art and Sketches, Ujjal Kr Dutta tackles the challenge of creating a single embedding model capable of spanning photos, art, cartoons, and sketches simultaneously. Dutta introduces a "Fuse and Attend" mechanism embedded within a contrastive learning framework. By pulling representations of the same semantic concept across different domains closer together while pushing distinct concepts apart, the model learns style-agnostic features that transcend the visual boundaries of any single medium.
Alternatively, instead of altering the network architecture, one can bridge the gap by synthesizing stylized training data. The authors of Improving Object Detection in Art Images Using Only Style Transfer demonstrate that applying arbitrary style transfer to standard photographic datasets (creating a stylized corpus called StyleCOCO) allows standard object detection models to adapt to non-photorealistic textures without losing their structural grounding.
The Logic of Abstraction and Semantic Grounding
The non-photorealistic domain gap is not merely a low-level visual mismatch; it also represents a failure in high-level abstract reasoning. Stylized media often violate real-world physical semantics, spatial alignments, and geometric proportions.
For instance, in Abstract Visual Reasoning with Tangram Shapes, Anya Ji et al. show that massive pre-training on natural images does not automatically enable the logic of abstraction required for human-like visual reasoning. When models like CLIP are presented with highly stylized, non-iconic forms like tangrams, they fail because they cannot map the abstract geometry to real-world concepts. However, the authors discover that fine-tuning these models on part-level descriptions allows them to learn the compositional logic of abstraction, bridging the gap between literal perception and symbolic reasoning.
This structural breakdown is also evident in sequential artistic media like comic books. As Emanuele Vivoli et al. point out in CoMix: A Comprehensive Benchmark for Multi-Task Comic Understanding, models optimized for natural photography struggle to parse multi-panel layouts, varying line weights, and shading conventions. This highlights that overcoming the non-photorealistic domain gap requires models to move beyond pixel-level feature matching and instead master the underlying structural and narrative grammar unique to creative expression.
Key Foundational Work
The Cross-Depiction Problem: Computer Vision Algorithms for Recognising Objects in Artwork and in Photographs (2015) — contributed 4 concept elements to this direction.