Models pretrained on real photographs and then fine-tuned on rendered images often get worse on real images. Accuracy on the synthetic data climbs toward perfect while real-world accuracy collapses. The usual explanation is that the network "latches onto simulator shortcuts," such as overly clean textures or missing high-frequency noise. That explanation names the failure but does not predict it. It cannot say how strong a shortcut must be before the model abandons what it learned on real data. It cannot say why a better pretrained starting point sometimes protects the model and sometimes does not. And it cannot say why a fix like regularizing toward the pretrained model works on some runs and not others. This note proposes a solvable model that answers these questions. It is a "spiked" random landscape: a random high-dimensional loss surface with two planted directions. One direction is the transferable real feature. The other is a shortcut that exists only in the simulator. We analyze it with the Kac-Rice formula, which computes the expected number and type of critical points of a random function at each loss level.
The core prediction is a phase transition. Below a critical shortcut strength, fine-tuning from a pretrained start stays in the basin around the real feature, no matter how long it trains. Above that strength, the basin loses stability and gradient descent leaves for the shortcut. Real accuracy should therefore fall off a cliff rather than decline smoothly. The location of the cliff should move in a computable way with three things: how well the pretrained start already aligns with the real feature, the learning rate, and the strength of any pull back toward the pretrained model. We test this on the standard synthetic-to-real benchmarks. There, the strength of a high-frequency shortcut can be dialed continuously with the amplitude perturbations of PASTA.
Core Claim
Two empirical findings look contradictory, and both are real. What is being transferred in transfer learning? shows that fine-tuning from a pretrained model stays in one basin: independently fine-tuned copies connect by a straight line in weight space with no loss barrier. Automated Synthetic-to-Real Generalization shows that fine-tuning an ImageNet model on VisDA-17 or GTA5 overwrites real-world features. Plain training with a large learning rate reaches only 28.2% real accuracy unless a distillation-style "proxy guidance" loss ties the model to a frozen ImageNet teacher.
Our claim is that these are two sides of one boundary in a phase diagram. Whether a run stays or escapes depends on whether the pretrained basin still contains a stable minimum under the synthetic objective. In a model with interactions of order three or higher, that is a Kac-Rice question with a sharp answer. Order matters for a concrete reason. In the order-2, matrix-like case, gradient descent eventually finds the largest planted direction regardless of where it starts. Only in the higher-order, tensor-like case does the starting overlap create a separate basin that can either survive or vanish. This is the regime where metastability (getting stuck in a basin that is not the global best) and initialization dependence appear in the analyses of spiked tensor landscapes by Sarao Mannelli et al. and Bonnaire et al..
The Model
Weights $w$ live on a sphere of radius $\sqrt{N}$. The synthetic training loss is
$$L_{\text{syn}}(w) = H_0(w) - \lambda\,N\left(\tfrac{w\cdot u}{N}\right)^p - s\,N\left(\tfrac{w\cdot v}{N}\right)^p .$$
Each term has a specific meaning:
- $H_0$ is an isotropic Gaussian random field of degree $p\ge 3$. It stands for the label-irrelevant roughness of the loss.
- $u$ is the transferable feature. It has strength $\lambda$ in both synthetic and real data.
- $v$ is a unit-norm shortcut direction orthogonal to $u$. It has strength $s$ in synthetic data and strength zero in real data.
The real loss is the same expression with $s=0$. Fine-tuning starts from a point with overlap $m_0 = w_0\cdot u/N$, the abstraction of pretraining.
The theoretical work extends the single-spike Kac-Rice computations in Bonnaire et al. and Ghio et al. to two spikes. We compute the annealed complexity, meaning the exponential growth rate of the expected number of critical points. We condition on two quantities: the overlaps $(m_u, m_v)$ with the two planted directions, and the Hessian index, meaning the number of downhill directions at the critical point.
The quantity of interest is the critical shortcut strength $s^*(\lambda, m_0)$. At $s^*$, the last family of minima with large $m_u$ and small $m_v$ disappears and is replaced by saddles that point toward $v$. Fan et al. and Maillard et al. compare annealed counts with quenched counts, which average the logarithm of the number of critical points and so reflect a typical sample rather than rare ones. Following them, we verify that the annealed threshold agrees with direct gradient-descent simulations near the transition.
Two further mechanisms are modeled explicitly. Proxy guidance enters as a quadratic pull toward $w_0$ with strength $\gamma$. This adds a restoring term that should raise $s^*$ by an amount we can compute. Large learning rates enter as effective noise in Langevin dynamics, which lets the optimizer cross barriers before the basin fully disappears. This should lower the effective threshold, which is consistent with ASG's observation that the large-learning-rate setting is where collapse is worst.
The model is deliberately crude. It inherits the isotropy assumptions that the research file flags as the main limitation of this literature, and it says nothing quantitative about ResNets. Its value is that it makes qualitative predictions about shape that can be falsified: a cliff rather than a slope, a threshold that moves with the pretrained overlap $m_0$ and with $\gamma$, and a barrier that jumps exactly at the cliff.
Research Program
The program has three stages, each anchored to the one before.
Theory. We derive $s^*(\lambda, m_0, \gamma)$ and the Hessian signature at the threshold. When the pretrained basin loses stability, the smallest Hessian eigenvalue at the basin minimum should cross zero, with its eigenvector aligned to $v$. This is the analogue of the outlier-eigenvalue detachment at the BBP transition (the point where a planted signal produces an eigenvalue that separates from the noise spectrum) that Bonnaire et al. describe for single-spike landscapes. We check the derivation against gradient descent on the model with $N$ between 500 and 10,000.
Controlled networks. We train ResNet-50 classifiers pretrained on ImageNet on VisDA-C synthetic renderings, as introduced in VisDA. We vary shortcut strength in two ways:
- Inverting PASTA: we suppress high-frequency amplitude variance in the renderings by a factor $\alpha$, making the simulator signature stronger.
- Planting a label-correlated high-frequency texture in the synthetic images with a tunable correlation, which gives a ground-truth shortcut.
We vary $m_0$ through the quality of the starting point: random initialization, a weakly pretrained checkpoint, and full ImageNet pretraining. We vary $\gamma$ through the weight on ASG's proxy-guidance loss.
Segmentation. We repeat the core sweep for GTAV→Cityscapes segmentation, using the FCN-Res50 setup from ASG and the DeepLabv3+ setup from PASTA.
For every run we record four things:
- real target accuracy or mIoU;
- the loss barrier along the straight line from $w_0$ to the fine-tuned weights, which is Neyshabur et al.'s probe, now used as an order parameter;
- the smallest eigenvalues of the synthetic-loss Hessian along the fine-tuning trajectory, computed with a Lanczos method;
- how much of each low eigenvector's effect on the network's outputs comes from high-frequency input content versus phase-preserved content.
All but the first are computed on synthetic data alone.
Baselines
Source-only fine-tuning. The naive reference point is plain fine-tuning of the pretrained model on synthetic data. ASG reports 28.2% VisDA-17 accuracy for this with a large learning rate, and 22.17% mIoU for FCN-Res50 on GTA5→Cityscapes. The no-adaptation AlexNet number in VisDA is 28% on COCO. We rerun source-only with our exact backbones and schedules, so every comparison uses identical data, epochs, and a hyperparameter search of the same size.
Proxy guidance. For classification, the strongest existing method on the question of staying in the pretrained basin is ASG's proxy guidance. It reaches 58.7% on VisDA-17 alone and 61.1% with its reinforcement-learning-tuned learning rates. Proxy guidance is the quantity $\gamma$ in our model, so it is both a competitor and an experimental knob. We reimplement it with the same tuning budget as every other arm.
PASTA. For segmentation, the strongest data-side method is PASTA. It reports moving GTAV→Cityscapes DeepLabv3+ from 28.95 to 44.12 mIoU, and +16.9 mAP on Sim10K→Cityscapes detection.
Flat-minimum seeking. We include Sharpness-Aware Minimization (SAM), which seeks flat minima. We also include the full-space random-perturbation loss of Flat-LoRA, which reports that flatness margins grow with distribution shift and costs only about 2% extra runtime.
What counts as beating the baselines. This is primarily an explanatory proposal, so "beating" a baseline means two things. First, the threshold predicted from synthetic-only Hessian measurements must locate the real-accuracy cliff in the $\alpha$ sweep more accurately than the competing synthetic-only predictors described below. Second, a prescriptive check: choosing $\gamma$ just above the predicted $s^*$ should match or exceed ASG's 58.7% on VisDA-17 without any real validation labels. If we fall short of 58.7% under the same budget, the prescriptive claim fails.
Key ablation. The ablation that isolates the theory's main ingredient removes the overlap axis. We fix shortcut strength and vary only the pretrained start. If the cliff location does not move with $m_0$, the basin-stability account is wrong, even if the accuracy numbers look good.
Why not just…?
Why not just sweep PASTA's strength using a small labeled real validation set? In practice this is cheap and will often work. We do not claim to beat it at picking hyperparameters when real labels exist. What it cannot do is say when augmentation alone is insufficient. Our model predicts that when the pretrained overlap $m_0$ is low, no amount of shortcut reduction short of $s=0$ keeps the run in the real-feature basin, and only the restoring term $\gamma$ helps. The $m_0$-by-$\alpha$ grid tests this directly. If augmentation strength alone recovers performance at low $m_0$, that part of the theory is refuted.
Why not just use flatness (SAM or Flat-LoRA) as the fix and as the predictor? In our model, the shortcut minimum can be just as flat as the real-feature minimum. Flatness measures curvature at the end point, not which basin was reached. The model therefore predicts that sharpness at convergence will correlate poorly with real accuracy across the $\alpha$ sweep, while the linear barrier from $w_0$ and the Hessian eigenvector alignment will track it. SAM and Flat-LoRA are run as full arms. If they close the gap as well as proxy guidance, the basin-escape story adds little and we will report that.
Why not just measure how sensitive the synthetic-trained model is to high-frequency input perturbations? This is the most serious threat to the diagnostic part of the proposal. A model that relies on a high-frequency shortcut will be visibly sensitive to high-frequency noise, and that check needs no Hessian. We include it as a head-to-head predictor of real accuracy. If it predicts the cliff as well as the Hessian-based signal, we drop the diagnostic claim. We would then keep the theoretical contribution: an explanation of why the decline is a cliff, why it moves with pretraining, and how much proxy guidance is enough. This is something input-sensitivity scores cannot provide. The planted-texture experiment matters here because it is the one setting where we know the true shortcut direction and can check which signal finds it first.
Why not just apply alignment methods like MMD or CORAL, or translate images with a GAN? These need unlabeled real data, and pixel translation PixelDA is limited to low-level style changes. Our question is about generalizing from synthetic data alone with no real data, so these are outside scope rather than competitors.
Proposed Experiments and Decisive Checks
Check 1: shape of the accuracy curve. On VisDA-C with the planted texture and the inverted-PASTA sweep, real accuracy as a function of shortcut strength should show a knee, where accuracy falls much faster over a narrow range than elsewhere. A roughly linear decline is the falsifying outcome. At the knee, the barrier along the straight line from the pretrained weights should jump from near zero to clearly positive. This is the transition from Neyshabur et al.'s basin staying to ASG's forgetting, observed in a single sweep.
Check 2: movement of the knee. The knee should shift toward larger shortcut strength as the pretrained overlap $m_0$ increases and as the proxy-guidance weight $\gamma$ increases. The ordering of shifts should match the two-spike phase diagram computed in theory, which is a check on direction and ordering rather than exact values.
Check 3: a synthetic-only early warning. On the synthetic loss, the smallest Hessian eigenvalue along the trajectory should approach zero just before the escape. Its eigenvector should be dominated by high-frequency-sensitive directions. This gives a warning computed without real labels, and we compare it against the input-sensitivity and sharpness predictors on the same runs.
Check 4: segmentation. We repeat Checks 1 through 3 for GTAV→Cityscapes using the published FCN-Res50 and DeepLabv3+ protocols. The anchors are the reported 22.17 mIoU for source-only and 29.64 for ASG with FCN-Res50, and 28.95 and 44.12 mIoU for baseline and PASTA with DeepLabv3+.
What would sink the proposal. The central account fails if any of the following holds: the decline is smooth, the knee does not move with $m_0$, or the barrier does not jump at the knee.
References
- Pasta: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization
- What is being transferred in transfer learning?
- Automated Synthetic-to-Real Generalization
- Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Models
- High-dimensional non-convex landscapes and gradient descent dynamics
- TAP free energy, spin glasses and variational inference
- Landscape Complexity for the Empirical Risk of Generalized Linear Models
- VisDA: The Visual Domain Adaptation Challenge
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks