Image classifiers trained on photographs recognize a dog by its fur long before they recognize it by its outline. That is why the same network that works on photos collapses on sketches, cartoons and paintings. The usual fix is to degrade texture during training, by scrambling colors or style-transferring photos into paintings. That helps on sketches but applies the same push toward shape everywhere, including cases where texture is the real signal. We propose a different route. Render training scenes synthetically, which yields free, perfect geometry (depth, surface normals, and wireframe edges) alongside every image. A large "teacher" network learns, token by token, how much to consult that geometry. A small photo-only "student" is then trained to imitate the teacher. At deployment the geometry, the renderer and the teacher are all discarded, so the student costs exactly as much as an ordinary small classifier.
The idea borrows from three places. Privileged information is shown only during training, as in skeleton action recognition that uses RGB and a language model as training-only teachers. The learned gate comes from gated cross-attention. Distillation from a fused teacher into cheap unimodal students comes from sketch retrieval. The new question is whether a gate trained on synthetic geometry can teach a small real-image model when to trust shape. The target is a model that gains on sketches and art without paying the known texture penalty on tasks where texture matters.
Core Claim
Consider a cross-modal attention teacher in which image tokens query synthetic geometry tokens. "Cross-attention" means one stream's features act as queries that retrieve and weight information from another stream. Wrap it in a learned per-token gate, and distill its gated features into a unimodal RGB student. We claim this student will match or exceed texture-suppression methods on non-photorealistic test domains, and lose less accuracy on texture-dependent recognition. The mechanism matters. The gate decides, for each image region, whether the class decision should draw on geometry or on appearance. Distilling the gated features transfers that decision policy into the student's classification path. Two simpler routes do not transfer it: auxiliary depth prediction adds geometric features the classifier may ignore, and uniform texture destruction removes appearance everywhere.
Concretely, for image token $h_i$ and geometry tokens $K_g, V_g$, the teacher computes
$$z_i = g_i \cdot \mathrm{softmax}\!\left(\frac{q_i K_g^\top}{\sqrt{d}}\right) V_g + (1-g_i)\, h_i ,$$
where $q_i$ is the query projection of $h_i$, $d$ is the key dimension, and $g_i \in [0,1]$ is a gate predicted from $h_i$. This is the gating form of An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention, which introduced it to stop one stream from overwhelming the other and to preserve unimodal feature identity. The student sees only the image, producing tokens $s_i$. It is trained with
$$\mathcal{L} = \mathcal{L}_{\text{CE}} + \lambda \sum_i \lVert \phi(s_i) - z_i \rVert^2 ,$$
where $\phi$ is a small projection used only in training and $\lambda$ weights the distillation term. On real photographs, which have no geometry, only the classification term applies.
Where This Sits
Three results motivate the proposal. Detecting People in Artwork with CNNs showed that textures break across art styles while global layout survives. It exploited this by freezing early layers, whereas we try to teach layout-reliance directly. Domain Randomization for Scene-Specific Car Detection and Pose Estimation showed that coarse CAD proxies with randomized textures train detectors that rely on geometry. Diversity is Definitely Needed showed that synthetic data need diversity more than photorealism. Both of these used the synthetic RGB image alone and threw away the geometry the renderer produced for free. We keep it.
Three texture-suppression methods are our main competitors. Robust and Generalizable Visual Representation Learning via Random Convolutions (RandConv) scrambles local texture with random filters. Towards Domain Adversarial Methods to Mitigate Texture Bias treats edge maps and stylized images as an adversarial target domain. Improving Object Detection in Art Images Using Only Style Transfer (StyleCOCO) style-transfers COCO. All three push toward shape uniformly. RandConv's authors state that this hurts texture-critical tasks, and that admission is the constraint our gate is designed around.
Two papers supply the method. Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition uses RGB video and a language model only at training time and ships a bare skeleton network. It works on real photographic video with contrastive alignment and has no gate. Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrieval (XModalViT) distills a sketch-photo cross-attention teacher into independent encoders. It needs real human sketch-photo pairs and targets retrieval, not robustness of a photo classifier.
Fuse and Attend (RCERM) also uses gated fusion on PACS, but across image domains within one backbone and without synthetic geometry or compression. Bridging Domain Generalization to Multimodal Domain Generalization (URMMDG) needs all modalities at test time. Neither transfers a policy for when to use shape into a cheap model.
Research Program
Synthetic data. We build a domain-randomized rendering set covering the PACS categories and people. Each rendered image comes with depth, normals and wireframe edges. The rendering has one controlled knob, $p$: the fraction of renders with randomized textures. The remainder get class-consistent materials. This knob is essential. If every synthetic texture were random, the teacher's gate would learn to always open toward geometry and would teach nothing beyond uniform shape bias. With $p<1$, appearance is sometimes informative, and the gate must learn when.
Teacher training. The teacher is a ViT-scale image encoder plus a lightweight geometry encoder, fused by the gated cross-attention above. It is trained on the synthetic images for classification. We randomly drop the geometry stream during this training so the teacher cannot become geometry-only. Following the collapse and feature-drift observations in the gated cross-attention paper, we also run an ungated softmax variant and a sparsemax variant.
Student training. The student is a ResNet-18. It trains on synthetic images with classification plus token distillation, and on real source photos with classification only.
Diagnostics. We then check that the gate behaves as claimed. It should open more on sketch-like synthetic inputs and on renders with randomized textures, and close on renders whose class-consistent materials are discriminative. Mean gate value should rise with $p$. This mechanistic check stands apart from accuracy.
Baselines
The naive reference is a ResNet-18 trained with cross-entropy on the same synthetic renders plus real source photos, with no geometry, augmentation or distillation. Every method below uses this same data, student architecture, epoch budget and a matched hyperparameter search of equal trial count. The teacher's cost is reported separately as training compute.
RandConv. The strongest single-stream competitor on the depiction gap is RandConv, applied to the same student with its mixing and consistency regularization. On PACS Sketch it reports a gain from roughly 55% to roughly 69%, and 76.11% with a RandConv-pretrained AlexNet. On ImageNet-Sketch it reports Top-1 improving from 10.28% to 18.09%. We rerun it rather than copy these figures, because the backbones differ.
Edge-map adversarial training. Towards Domain Adversarial Methods to Mitigate Texture Bias is the closest use of geometry-like signals and reports 11% higher ImageNet-R accuracy than a standard IN-200 model. We run it with our rendered wireframes as its edge domain, so it gets exactly the geometry our teacher gets.
Person detection. For People-Art, the reference is StyleCOCO fine-tuning at a reported AP50 of 0.68 (Kadish et al.), and fine-tuned VGG16 at 59% AP (Westlake et al.). Our detection variant uses the same detector with a distilled backbone.
Ablations isolating the gate. Four ablations isolate the key ingredient. The gate is fixed at one (pure geometry), fixed at zero (the distillation target is an image-only teacher, which is ordinary same-modality distillation), removed (ungated softmax), and the student is trained with an auxiliary depth-and-normal prediction head instead of distillation.
Win condition. Beating the baselines means the proposal must do three things at once. It must exceed RandConv and edge-adversarial training on average Art/Cartoon/Sketch accuracy under single-source-photo PACS. It must stay within one point of the naive model on in-domain photos. It must lose significantly less than RandConv on a texture-critical fine-grained benchmark. For that texture-critical benchmark, no reported result was found on Lacuna for any method in this file, so we establish all numbers ourselves. Gains on sketches that come with RandConv-sized texture losses would not count.
Why not just…?
Why not just add an auxiliary head that predicts depth and normals from the rendered RGB? This is the most serious threat, because it uses the same free geometry at zero inference cost with far less machinery. Our argument is that it enriches features without changing what the classifier relies on, and applies no notion of "only where useful." But we do not assume this. It is a head-to-head ablation with identical data. If it matches our method on both the depiction and texture-critical axes, the honest conclusion is that the gate adds nothing, and we would report the auxiliary head as the recommended recipe.
Why not just feed the edge and normal renders as extra training images, or use them as an adversarial domain? Kashyap et al. do essentially this and gain on ImageNet-R. It applies shape pressure uniformly, so we expect it to share RandConv's texture penalty. The texture-critical evaluation and the gate-fixed-at-one ablation are designed to show whether that happens.
Why not just use RandConv or StyleCOCO and skip multimodal training? These are cheap and effective on sketches. RandConv's own authors report losses where texture discriminates, and StyleCOCO swaps textures without supplying geometry. They are also orthogonal to our method, so we additionally test our method combined with RandConv. If that combination is simply additive, the gate contributes something RandConv lacks.
Why not just use CLIP or distill from a foundation model? A ResNet-50 trained on diverse Stable Diffusion data already beats CLIP-ResNet50 on CIFAR-10, 81.0% versus 75.6% (Shipard et al.). Abstract Visual Reasoning with Tangram Shapes reports off-the-shelf CLIP at only 15–18% on abstract tangrams. Foundation models are also too large for the edge setting that motivates this work. We still include a CLIP-feature distillation baseline into the same student, because it is the natural "bigger teacher" alternative to a geometry teacher.
Why not just fuse RGB and geometry at test time with concatenation or late fusion? Real test images have no rendered geometry. Even if geometry were estimated, running two backbones reproduces the overhead that MultiBench documents, where multimodal transformers can need over 12 times the inference time of unimodal models.
Proposed Experiments
The primary benchmark is PACS under single-source training on Photo plus our renders, tested on Art, Cartoon and Sketch. Secondary benchmarks are ImageNet-Sketch and ImageNet-R on the classes our renders cover, and People-Art for detection. Every student reports parameters, FLOPs and measured latency to confirm they are identical to the naive student.
Four checks are decisive. The first sweeps the texture-randomization rate $p$. If the gate is doing its job, mean gate value should rise with $p$, and the student's sketch gains should persist while its texture-critical accuracy stays flat. A gate that stays near one regardless of $p$ would mean we have built an expensive RandConv.
The second compares gated, ungated and sparsemax teachers, measuring the cosine similarity between fused and original image tokens over training. This tests whether the feature-drift finding from the drug-target interaction paper recurs when synthetic geometry is much cleaner than photographic texture.
The third probes shape reliance directly. We evaluate every student on DexiNed edge maps and AdaIN-stylized versions of PACS photos, which are the augmentations Kashyap et al. used. The gated student should rely more on shape than the image-only-teacher ablation on these shape-only probes, while matching it on unaltered photos.
The fourth is the auxiliary-head comparison. It decides whether cross-modal attention earns its place here, or whether synthetic geometry alone, used the simple way, is the real finding.
References
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Detecting People in Artwork with CNNs
- Domain Randomization for Scene-Specific Car Detection and Pose Estimation
- Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion
- Robust and Generalizable Visual Representation Learning via Random Convolutions
- Towards Domain Adversarial Methods to Mitigate Texture Bias
- Improving Object Detection in Art Images Using Only Style Transfer
- Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition
- Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrieval
- Fuse and Attend: Generalized Embedding Learning for Art and Sketches
- Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representations
- Abstract Visual Reasoning with Tangram Shapes
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning