Simulators hand out pixel-perfect object masks for free, yet models trained on synthetic images still collapse on real ones. In the VisDA challenge, an AlexNet that reaches 99% on rendered CAD objects falls to 28% on real COCO photos (VisDA). The masks are an obvious way to tell a vision transformer (an image model that splits a picture into patches and lets a special summary token, [CLS], gather information from them through attention) where the object is. The obvious version of this idea has a hidden flaw. A model can be trained to put its attention weights on object patches without actually using those patches. It can shrink the vectors it reads from object patches. It can pull its decision from a few extra "register" tokens. Or it can coast on information carried forward by skip connections. Each of these escape routes is documented in prior analysis work. Supervising attention weights alone therefore risks producing nice-looking heatmaps on top of an unchanged shortcut.
This note proposes to supervise what the summary token actually receives instead of what it looks at. We measure how much of each layer's update to [CLS] comes from object patches, as opposed to background patches, registers, and the residual stream. We train with a loss that pushes that share up on synthetic images, using the simulator's masks. The core claim is that weight-level supervision is gameable and contribution-level supervision is not. Only the latter should change the model's causal reliance on objects, and with it real-domain accuracy. We test this directly on real images, using COCO's own masks to swap backgrounds and see whether predictions move.
Core Claim
Two analysis papers motivate the proposal by showing that attention weights misstate information flow. Attention is Not Only a Weight shows that a token's real contribution to an attention output is the norm $\|\alpha f(x)\|$, not the weight $\alpha$. Tokens like [SEP] receive large weights but carry tiny vectors. Switching from weights to norms lowers translation alignment error from 29.8 to 25.0 AER. Register and [CLS] tokens induce a decoupling shows that in large ViTs the summary token draws its class signal from registers or skip connections. In Giant DINOv2, the features implied by patch attention are nearly orthogonal to the actual output (cosine −0.0092).
Supervised-attention work in NLP is constrained by this. Stacey et al., HateXplain and Jayaram & Allaway all penalize scalar attention weights against human rationales. HateXplain itself flags the resulting plausibility–faithfulness gap.
We make a falsifiable prediction. Under synthetic-to-real shift, weight-level mask supervision will raise attention-on-object scores on real images while leaving background sensitivity largely intact. Contribution-level supervision will reduce background sensitivity and improve real accuracy. If the two behave the same on our counterfactual test, the norm/register concern is not practically important and the simpler method should be preferred.
The Supervised Quantity
For layer $l$ and head $h$, let $\alpha_j$ be the weight [CLS] places on token $j$ and $f(x_j) = W_O W_V x_j$ the vector that token contributes, following Kobayashi et al. Let $O$ be the patches whose simulator mask overlaps the object above a threshold, and $r^{(l)}$ the incoming residual [CLS] vector. The object share is
$$s^{(l)} = \frac{\sum_h \sum_{j \in O} \|\alpha_j f(x_j)\|}{\sum_h \sum_{j} \|\alpha_j f(x_j)\| + \|r^{(l)}\|},$$
where the inner sum over all $j$ includes background patches and registers. The training loss on synthetic images is cross-entropy plus $\lambda \sum_{l \in L} \max(0, \rho - s^{(l)})$. Here $L$ is a set of supervised layers, and $\rho$ is a target share below 1, so the model keeps some room to read context and registers.
Including $\|r^{(l)}\|$ in the denominator is deliberate. It closes the skip-connection escape route, because the model cannot satisfy the loss by freezing an early, background-laden [CLS] and attending to objects only in name. The hinge avoids forcing all-object reading. That forcing would hurt if registers legitimately store global information, as Vision Transformers Need Registers argues.
Per-token norms ignore LayerNorm and the MLP, which is a known approximation from Kobayashi et al. The counterfactual evaluation below exists precisely so that we do not have to trust this proxy.
Research Program
We fine-tune an ImageNet-pretrained ViT-B/16, with and without four registers, on VisDA-C synthetic renders. Object masks come from the render alpha channel or re-rendering of the CAD models. The file does not say whether VisDA ships masks, so we derive them. We evaluate on the real COCO and YouTube-BB targets.
The real-domain faithfulness test uses COCO's instance masks. For each real test image we create three variants: the object pasted onto another image's background, the background kept with the object patches blurred, and the object kept with the background replaced by noise. The metric is the prediction flip rate under background swap. A model that truly reads the object should be stable under that swap and should collapse when the object is removed.
A second testbed is synthetic-to-real LineMod pose estimation, where render masks are trivially available and PixelDA gives a reference. Layer and head selection follow Stacey et al.'s finding that supervising 3 of 12 heads beat supervising all. We sweep supervising all heads against greedy top-$K$ heads, and the last 4 layers against all layers. This tells us whether partial supervision simply relocates the shortcut to unsupervised heads.
Baselines
The naive reference is a source-only ViT fine-tuned on VisDA-C synthetic data with cross-entropy, using the same augmentation, schedule, and early stopping on a held-out synthetic split. No real labels are used for any model selection. For orientation, the only source-only number in the file is AlexNet at 28% on real COCO (VisDA). No reported result was found on Lacuna for a ViT source-only on VisDA-C, so we produce it ourselves.
The key ablation is weight-level mask supervision, a direct transfer of Stacey et al.'s MSE between [CLS] attention weights and a normalized mask. It uses the same layers, heads, and $\lambda$ grid as our method. This isolates the single ingredient we claim matters, which is supervising contributions rather than weights. Beating it means lower background-swap flip rate and higher real accuracy at matched synthetic accuracy.
The strongest same-information competitors are two methods that consume the identical masks without touching attention. The first is an auxiliary per-patch segmentation head, in the spirit of Sim2Seg's use of canonical segmentation. The second is background randomization: compositing masked synthetic objects onto random real or textured backgrounds. Each gets the same tuning budget, 12 hyperparameter configurations per method across three seeds.
We also run registers alone (Darcet et al.) and a ViT adaptation of Attention Diversification. The latter is the closest prior work that regularizes attention for domain shift, but it spreads attention by entropy rather than concentrating it with masks. Its reported ResNet-50 numbers (88.2% on PACS, 71.4% on Office-Home) are on different benchmarks and are not comparable here.
Finally, we report unsupervised domain adaptation as a separate track, since it uses unlabeled real images. This includes the VisDA-reported DAN at 51.62% and the challenge winner's pseudo-labeling pipeline at 92.8%. We do not claim to beat 92.8% source-only. The claim is that our loss adds on top of a pseudo-labeling baseline when both are run with identical target data.
Why not just…?
Why not just randomize backgrounds with the masks? This is the most serious threat. It removes the background shortcut at the data level without any attention-level supervision. It cannot, however, stop the model from reading spurious on-object rendering cues or register-stored global statistics, and it changes the training distribution rather than the routing. The experiments settle it directly: if background randomization alone matches our method on real accuracy and flip rate, we will report that the attention loss is unnecessary. We also run the combination, because the two may be complementary.
Why not just add a segmentation head? A segmentation head forces patch features to encode objectness. It says nothing about which patches [CLS] reads. The decoupling result (Register and [CLS] tokens; Register and [CLS] tokens induce a decoupling of local and global features in large ViTs) suggests patch quality and classification routing can come apart. Our prediction is good real-domain segmentation from this baseline alongside an unchanged background-swap flip rate. If that prediction fails, the routing argument is wrong.
Why not just supervise attention weights, as NLP did? Weight supervision is cheaper and already showed out-of-distribution gains on HANS (+1.59%) in Stacey et al.. But the norm analysis shows weights can be large on near-zero vectors, so the loss can be met cosmetically. We diagnose this directly by logging $\|f(x_j)\|$ on object patches under weight supervision. If those norms shrink, the gaming is confirmed.
Why not just use registers? Registers clean up attention maps. They raise VOC CorLoc from 35.3 to 55.4 (Darcet et al.). They also become the very sink through which decisions bypass patches. We run register and no-register backbones under every method to see whether supervision re-couples [CLS] to patches. Re-coupling would show up as the cosine between patch-implied and actual outputs rising from near zero.
Proposed Experiments and Decisive Checks
The primary result covers six configurations on VisDA-C: source-only, weight supervision, contribution supervision, segmentation head, background randomization, and background randomization combined with contribution supervision. For each we report real COCO and YouTube-BB accuracy, with three seeds and confidence intervals.
The decisive check has four parts. Contribution supervision must lower the COCO background-swap flip rate relative to weight supervision. Weight supervision must achieve comparable attention-on-object scores. Real accuracy must track flip rate rather than attention scores. And the gains must persist over the background-randomization baseline. If weight and contribution supervision tie on flip rate, the central claim fails.
A secondary check on LineMod measures mean angle error against PixelDA's reported 23.5°, which uses unlabeled target images; ours uses none. We also measure what fraction of PixelDA's gap our source-only method closes.
A mechanistic check sweeps the hinge target $\rho$. If accuracy peaks at intermediate $\rho$, that supports reconciling the diversify-versus-concentrate tension between Attention Diversification and mask supervision. In that reading, the model should mostly read the object, but not exclusively.
A natural extension is to GTA-V stencil masks for detection on KITTI. There the best file-reported synthetic-only number is 0.6803 mAP on Easy (Driving in the Matrix). We pursue it only if the classification results hold.
References
- VisDA: The Visual Domain Adaptation Challenge
- Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
- Register and [CLS] tokens induce a decoupling of local and global features in large ViTs
- Supervising Model Attention with Human Explanations for Robust Natural Language Inference
- HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
- Human Rationales as Attribution Priors for Explainable Stance Detection
- Vision Transformers Need Registers
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data
- Attention Diversification for Domain Generalization
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?