Models trained only on simulator renders often fit their training data almost perfectly and then fail on real images. In the VisDA challenge, an AlexNet reaches 99% accuracy on synthetic renderings of CAD models but only 28% on real COCO photos of the same twelve classes (VisDA). A natural remedy is sharpness-aware minimization (SAM), a training method that nudges the weights toward the worst nearby point and penalizes the loss there, so the model settles in a "flat" region where small weight changes do not hurt. For domain shift, though, plain SAM gives disappointing gains. On single-source PACS it reaches 64.8% against 74.7% for a method built specifically for shift (UDIM).
We propose that SAM underperforms on synthetic data for the same reason it underperforms on language models. When training loss is nearly zero, the cheapest way for SAM to shrink its penalty is to make the network's outputs more confident. Smoothing the input-to-output mapping, which is the effect SAM is meant to have, is the more expensive route and gets skipped. Avoiding spurious sharpness minimization broadens applicability of SAM identified this "confidence shortcut" in text models but never examined vision under domain shift. Our claim is that synthetic imagery behaves like text in this respect: it is easy to separate, quick to saturate, and full of rendering cues. If the claim holds, two things follow. Correcting SAM's update should help most exactly when the training source is synthetic. And a cheap measurement taken during training should predict how large the sim-to-real gap will be.
Core Claim
Let $z = f(\theta, x)$ be the network's output scores (logits) for weights $\theta$ and input $x$. Let $J = \partial z / \partial \theta$ be the network Jacobian and $g = \partial \ell / \partial z$ the gradient of the loss with respect to the logits. To first order, SAM penalizes
$$\rho \,\lVert \nabla_\theta \mathcal{L} \rVert = \rho \,\lVert J^\top g \rVert ,$$
where $\rho$ is the perturbation radius. There are two ways to make this small. The first is the logit path: shrink $g$ by pushing softmax outputs toward one-hot, which changes nothing about how the features respond to the input. The second is the functional path: shrink $J$ along the directions $g$ points to, which makes the mapping itself smoother. Avoiding spurious sharpness minimization showed that in language models these two components of SAM's update point in opposing directions and the logit path wins. Their fix, Functional-SAM, discards the logit-path component.
Our hypothesis has two parts. First, on synthetic sources the two components become anti-aligned early in training, much as they do for text, while on real-image sources of matched difficulty they stay aligned longer. Second, this anti-alignment explains why SAM transfers poorly from simulation to reality. The new content is not Functional-SAM itself. It is the claim that the source domain's statistics, and not only the input modality, decide which path SAM takes. We then test that claim with a controlled interaction experiment.
Relation to Prior Work
Three papers motivate the gap. They show that flatness-seeking helps under shift only when it is adapted. UDIM argues that flatness on the source does not carry over to unseen targets. It aligns the source loss surface with surfaces built from adversarially perturbed inputs, and it is the strongest single-source method in our file. DA-SAM probes curvature in multiple directions and scales the radius per domain, but it requires several source domains. That rules it out for the one-simulator setting. SAGM aligns gradients at the clean and perturbed points to avoid SAM's "sharpness trap." All three treat the failure of SAM as a problem of which neighborhood gets explored. None asks how SAM satisfies its own penalty. We do not change the neighborhood; we change which route through the network the penalty is allowed to take.
Two papers constrain the design by showing that flatness itself is not the target. Forget Sharpness finds that SAM's benefit comes from "forgetting" output biases. Its output-bias-forgetting variant lands in sharper minima yet improves ImageNet-R from 21.63% to 25.31%. SAM-ON perturbs only normalization parameters and likewise reaches sharper but more robust solutions. Both results fit our account. Both methods partly sidestep the logit path, so we include them as competitors rather than ignore them. Our evaluation therefore reports transfer accuracy, never flatness, as the outcome measure.
VLTSeg shows that a CLIP-pretrained backbone fine-tuned on GTA5 generalizes to real driving datasets better than elaborate domain-generalization methods. It credits language as a "semantic anchor" but never looks at optimization. We supply a testable mechanism for its finding: vision-language initialization may keep the logit and functional paths aligned during synthetic fine-tuning.
Research Program
The first stage is diagnostic. We train ResNet-50 and ViT-B/16 classifiers with plain SAM and track two quantities across training: the cosine between the logit-path and functional-path components of SAM's update, and the fraction of the penalty's decrease attributable to each. The decomposition follows the one in Avoiding spurious sharpness minimization. We run this under two sources with the same twelve classes. The synthetic source is VisDA-C's roughly 152k CAD renders. The real source is VisDA-C's roughly 55k COCO validation images. Both sources are tested on VisDA-C's held-out real YouTube Bounding Boxes frames, which gives a real-to-real control with an identical label space (VisDA). We compare at matched training accuracy, not matched step count, so that "synthetic is just easier" is controlled for.
The second stage is the intervention. We apply Functional-SAM, with and without the AdamW-preconditioned perturbation from the same paper, to synthetic-source training. We also test a cheap variant that applies the functional-only perturbation to normalization parameters alone, in the spirit of SAM-ON, to address the cost of doubled backpropagation.
The third stage is predictive. Across backbones (ImageNet versus CLIP initialization), radii and checkpoints, we test whether path alignment measured on the synthetic training set alone predicts the eventual synthetic-to-real accuracy drop. We compare it against source-only signals that are already used: the normalized Hessian trace from CR-SAM, and the per-sample curvature from Memorization Through the Lens of Curvature. A predictor of this kind matters in practice because in true sim-to-real work no real labels exist at training time.
Baselines
The naive reference is source-only empirical risk minimization, meaning ordinary training on renders with no special regularization. VisDA reports 28% real accuracy for AlexNet trained this way. We rerun it on our backbones, and we also give it twice the epochs so that it matches SAM's compute budget.
Plain SAM is the direct comparison the hypothesis is about, run with the same radius grid as every SAM variant.
UDIM is the strongest existing single-source method. It reports 74.7% on single-source PACS against SAM's 64.8% (UDIM). We run the authors' objective on VisDA-C and on PACS. Using the same backbones and an equal number of hyperparameter trials, we first reproduce its PACS number within a point and only then compare. SAGM and output-bias forgetting serve as the strongest alternative explanations of SAM's mechanism, and SAM-ON as the strongest cheap one, all under the same tuning budget.
For segmentation, the strongest reference is VLTSeg's full fine-tuning of an EVA-02-L-14 backbone on GTA5. It reports a 63.1% mean mIoU (mean intersection-over-union, averaged over classes) across Cityscapes, BDD100k, Mapillary and ACDC (VLTSeg). We do not expect a compact model to beat that number. We instead apply our optimizer on top of the same recipe at a smaller backbone scale. A win means a statistically reliable improvement in that four-dataset mean at fixed backbone, and in the best case closing part of the gap to the large model.
The key ablation separates the shortcut from the smoothing. We compare SAM with label smoothing, and SAM with an explicit logit-norm penalty, against Functional-SAM at matched final source-set confidence. If blocking overconfidence alone recovers the gains, then the functional path is not doing the work.
Beating a baseline means a gain over three seeds that exceeds seed variance, on VisDA-C real test accuracy and on PACS single-source accuracy.
Why not just…?
Why not just use label smoothing or temperature scaling to stop overconfidence? This is the most serious threat, because the logit path is literally a confidence channel. Our view is that these remedies cap the symptom: they make $g$ small for a fixed reason while leaving $J$ untouched, whereas Functional-SAM acts on $J$. But this is an empirical question, and the ablation above, run at matched confidence, settles it. If label smoothing plus SAM matches Functional-SAM, we will report that the sim-to-real failure of SAM reduces to calibration, which is a simpler result and still a useful one.
Why not just fine-tune a large vision-language backbone, as VLTSeg does? It works, at 63.1% DG mean, but it needs 304M parameters. It also leaves the brittleness of embodied models unexplained: AnyCamVLA reports pretrained vision-language-action policies falling to 67.9% success under small camera shifts. Our backbone comparison tests directly whether CLIP initialization helps because it keeps the two paths aligned. Our optimizer is orthogonal to the backbone, and we stack it on top of that recipe.
Why not just use UDIM's input perturbations, or domain randomization? UDIM is the strongest single-source competitor and may well win on PACS. It addresses a different failure, the mismatch between source and target loss surfaces, using 1D weight perturbations that could still take the confidence shortcut. We therefore also test UDIM with its weight perturbation replaced by the functional-only perturbation. If our account is right, the gains should add up.
Decisive Checks
The central test is an interaction effect. Functional-SAM's gain over SAM should be clearly larger with the synthetic VisDA-C source than with the real COCO source on the same classes and test set. Similarly, anti-alignment of the two paths should appear earlier for synthetic sources at matched training accuracy. If the gains are equal across sources, the "synthetic behaves like text" claim fails and only a generic optimizer improvement remains.
The second test is prediction. Path alignment measured on source data only should rank checkpoints and backbones by real-target accuracy better than the normalized Hessian trace does. We measure ranking quality with Spearman correlation over at least thirty runs.
The third test is the confidence-matched ablation, which decides whether the mechanism is smoothing the mapping or merely calibrating the outputs.
If all three tests hold, extending the diagnostic to the action outputs of vision-language-action policies on LIBERO is a natural next step. We leave it out of the core claim because the file contains no landscape measurements for such models.
References
- VisDA: The Visual Domain Adaptation Challenge
- Unknown Domain Inconsistency Minimization for Domain Generalization
- Avoiding spurious sharpness minimization broadens applicability of SAM
- Navigating the Flatlands: Dual Adaptive Sharpness-Aware Minimization for Domain Generalization
- Sharpness-Aware Gradient Matching for Domain Generalization
- Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics
- Normalization Layers Are All That Sharpness-Aware Minimization Needs
- Strong but Simple: A Baseline for Domain Generalized Dense Perception by CLIP-Based Transfer Learning
- CR-SAM: Curvature Regularized Sharpness-Aware Minimization
- Memorization Through the Lens of Curvature of Loss Function Around Samples
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models