When an ImageNet-pretrained network is fine-tuned on rendered images, such as computer-graphics models of cars or video-game street scenes, it gets very good at the synthetic data and much worse on real photographs. The network learns to rely on rendering quirks that never appear in real images, and in doing so it overwrites the general visual features it brought from pretraining. We propose a simple geometric account of what goes wrong and a regularizer that follows from it. The account is that the features the network computes for synthetic images begin to occupy new directions that the pretrained network never used, and these directions are where the rendering shortcuts live. The regularizer penalizes the fraction of each feature vector that falls outside the subspace spanned by the frozen pretrained network's features on the same synthetic images. Within that subspace the network may still rotate, rescale and recombine its features freely.
The setting is single-source synthetic-to-real generalization. Only labeled synthetic data is available during training and no real images are seen. This is the regime of Automated Synthetic-to-Real Generalization (ASG), which documents the forgetting problem and counters it with a prediction-level KL penalty toward a frozen ImageNet copy plus a reinforcement-learning controller for layer-wise learning rates. Our claim has two parts. First, a constraint on the span of features is a better-targeted way to prevent forgetting than constraining output predictions (ASG) or freezing the backbone. Second, the angular drift between the fine-tuned and pretrained feature subspaces, measured on synthetic data alone, predicts how badly the model will do on real data. That second part would give a label-free early-stopping and model-selection signal for a setting that currently has none.
Core Claim
Some terms first. A Grassmann manifold is the space of all $k$-dimensional linear subspaces of $\mathbb{R}^n$. Two subspaces are compared by their principal angles, which are the successive smallest angles between unit vectors drawn from each subspace. This geometry is well developed for comparing sets of observations. Jayasumana et al. build positive-definite kernels from projection distances, and Hong et al. observe that low-energy subspace directions are the unstable ones. Our use is narrower: the subspace is a reference span for a network's representation, not a description of a dataset.
Let $f_0$ be the frozen pretrained feature extractor and $f_\theta$ the one being fine-tuned. Both map an image to a penultimate-layer vector in $\mathbb{R}^n$, with $n = 2048$ for ResNet-50. Let $U \in \mathbb{R}^{n \times k}$ be an orthonormal basis for the top-$k$ principal subspace of $f_0$'s features on the synthetic training set. It is computed once by an SVD, which is cheap for a 2048-dimensional covariance. We add a span-leakage penalty to the synthetic classification loss:
$$ L_{\text{leak}}(\theta) = \frac{1}{|B|}\sum_{x \in B} \frac{\lVert (I - UU^\top)\,(f_\theta(x) - \mu_0)\rVert^2}{\lVert f_\theta(x) - \mu_0\rVert^2}, $$
Here $B$ is a mini-batch of synthetic images and $\mu_0$ is the mean of $f_0$'s features on synthetic data. Centering at $\mu_0$ addresses the first-order mean-shift issue that Liu et al. flag as unhandled by zero-mean Grassmann formulations. We will compare centered and uncentered versions directly.
The penalty is one-sided and invariant to scale. It is zero for any feature that stays in the pretrained span, however that feature has been reweighted or remapped. It grows only when energy moves into new directions. As a diagnostic we also track the geodesic drift $d(t) = (\sum_i \phi_i(t)^2)^{1/2}$. In this expression, $\phi_i(t)$ are the principal angles between $U$ and the top-$k$ subspace of $f_\theta$ at training step $t$, both computed on held-out synthetic images.
The mechanism differs from the closest prior work in specific ways. Invariant-feature Subspace Recovery (ISR) finds spurious directions from how class statistics drift across at least two labeled environments, then projects them out. Our setting has only one synthetic environment. We use the pretrained model as the second reference point, so the directions the synthetic fine-tuning adds play the role of ISR's spurious subspace. Rate–Distortion Invariance (RDI) aligns class subspaces across multiple source domains on the Grassmannian and controls their spectral volume, but it does not address forgetting of pretraining. Grassmannian adaptation methods such as the Geodesic Flow Kernel, progressive domain augmentation (PrDA) and graph-attentional landmark selection all need unlabeled target data and mostly operate on fixed features. Deep LogCORAL supplies the precedent that respecting curved geometry in a deep loss helps. It works on covariance matrices and needs an eigendecomposition every step; our anchor is computed once.
Research Program
The work has three parts.
The first part is the measurement study, and it is cheap. We fine-tune ImageNet ResNet-50 on VisDA-2017 synthetic data with standard cross-entropy, saving checkpoints. For each checkpoint we record $d(t)$ and the leakage fraction on synthetic data, alongside real-domain accuracy on the VisDA COCO validation set. The hypothesis is that real accuracy falls as drift rises, with a strong negative rank correlation across checkpoints and across learning-rate settings. Learning-rate settings matter here because ASG reports that large learning rates give 28.2%. If the hypothesis fails, meaning drift and real accuracy are unrelated, the geometric story is wrong and the regularizer has no reason to work. We would stop there.
The second part is the regularizer, $L_{\text{cls}} + \lambda L_{\text{leak}}$. We will study three questions. The first is the choice of $k$: whether a span capturing 90%, 95% or 99% of pretrained energy is right, since too small a span blocks needed features. The second is a global anchor versus per-class anchors computed from $f_0$ on each synthetic class. The third is whether applying the penalty at intermediate stages as well as the penultimate layer helps. We also test a soft version that down-weights low-energy anchor directions, following the observation of Hong et al. that these directions are unstable.
The third part is the transfer study. We use the regularized model as an initialization for unsupervised adaptation, as ASG did. This tests whether a span-preserved starting point also helps once unlabeled real data becomes available.
Baselines
The naive reference is source-only fine-tuning with cross-entropy on synthetic data. ASG reports 28.2% VisDA-2017 accuracy for ResNet-50 under a large learning rate and 22.17% mIoU for FCN-Res50 on GTA5→Cityscapes. We will rerun it under our own schedule rather than rely on those figures, and report both.
The strongest existing method in this exact setting is ASG. The file reports 58.7% VisDA accuracy with proxy guidance alone, 61.1% with the reinforcement-learning learning-rate controller, and an improvement from 22.17% to 29.64% mIoU on GTA5→Cityscapes. We will reimplement proxy guidance (KL toward the frozen ImageNet model) with the same backbone, data, epochs and hyperparameter search budget as ours. The full controller result we will quote as reported. If our reimplementation of proxy guidance falls well short of 58.7%, we will report that, because it bounds how much of any gap is attributable to our pipeline.
Beating ASG means exceeding 61.1% on VisDA and 29.64 mIoU on GTA5→Cityscapes without a learned controller. Matching proxy guidance's 58.7% at lower complexity would be a weaker but still useful result. Hyperparameters, meaning $\lambda$, $k$, and the KL weight for the baseline, are selected on held-out synthetic data only. We also report oracle selection on real validation data separately, so the reader can see how much each method depends on peeking at real labels.
Three further baselines isolate the key ingredient, which is constraining the span rather than the features or the outputs. Feature-level L2 distillation to $f_0$ penalizes any change to the feature vector, including movement inside the span. No reported result was found on Lacuna for feature distillation on VisDA. Linear probing on the frozen backbone keeps the span exactly but allows no change at all. No reported result was found on Lacuna for a linear probe on VisDA-2017. A random-subspace control replaces $U$ with a random $k$-dimensional subspace of matched dimension and tests whether the pretrained span specifically matters.
Unsupervised adaptation methods are listed only as references from a different, easier setting, because they see real images. These are DAN at 51.62% and the challenge-winning Mean Teacher at 92.8% (Peng et al.; French et al.). For the initialization study, the comparison point is ASG's reported 84.6% when used as a UDA initialization.
Why not just…?
Why not just distill features with an L2 penalty to the frozen model? This is the most serious threat. If L2 feature distillation matches the span penalty, our "span, not values" argument is empty and the contribution reduces to the drift diagnostic. The case for the span penalty is that classification on new synthetic classes needs features to be reweighted, which L2 forbids and the span penalty allows. That argument is only a hypothesis. The head-to-head comparison at equal tuning budget settles it. We will also report synthetic training accuracy to show whether L2 hurts by underfitting.
Why not just use ASG's KL proxy guidance? KL on outputs constrains only what the frozen model's 1000-way ImageNet head can see. Shortcut directions that do not change ImageNet logits pass through unpenalized. ASG itself notes that proxy guidance depends on the target resembling ImageNet (Chen et al.). The measurement study will show whether KL-regularized runs still accumulate span leakage. Combining KL with the span penalty tests whether the two are complementary or redundant.
Why not just freeze the backbone and train a linear head, or lower the learning rate? Freezing keeps the span perfectly, so if span leakage were the whole story, linear probing would be enough. We expect it to underfit, because the useful synthetic-to-real features must be recombined nonlinearly. But that is an empirical question that the linear-probe baseline answers directly. A sweep over small uniform learning rates is the cheap version of ASG's controller. The drift diagnostic should explain why a given learning rate works, and our regularizer must beat the best learning rate in that sweep.
Why not just align covariances (Deep CORAL or LogCORAL)? Those methods need real target features, which this setting forbids. Liu et al. also report that CORAL falls below plain empirical risk minimization in low-data regimes. They are out of scope as competitors but relevant if the method is extended to adaptation.
Proposed Experiments
The primary benchmark is VisDA-2017 classification: about 152k synthetic CAD renderings for training, with the real COCO crops for validation, reporting mean per-class and overall accuracy. The secondary benchmark is GTA5→Cityscapes segmentation with FCN-Res50, reporting mIoU, where the penalty is applied to backbone features pooled over pixels. Both use the ImageNet ResNet-50 initialization that ASG uses. Each configuration runs with three seeds.
The first decisive check is the drift–accuracy correlation from part one, computed across checkpoints and learning rates. We want a Spearman correlation strongly negative and stable across seeds. We will also check whether drift, used as a stopping rule, picks checkpoints within a point or two of oracle real-data selection.
The second decisive check is the comparison of the span penalty against L2 feature distillation, linear probing, random-subspace control and proxy guidance at matched budget on both benchmarks. The idea survives only if it beats L2 distillation and linear probing, and if the random subspace gives no benefit.
The third decisive check is whether the regularized model, used as a UDA initialization, approaches or exceeds ASG's reported 84.6%.
A negative outcome on any of the first two checks is informative and will be reported as such. If drift does not predict real accuracy, forgetting is not a span phenomenon. If L2 distillation matches the span penalty, the freedom to rotate within the span buys nothing.
References
- Automated Synthetic-to-Real Generalization
- Kernel Methods on Riemannian Manifolds with Gaussian RBF Kernels
- Disturbance Grassmann Kernels for Subspace-Based Learning
- Geometric Rate–Distortion Invariance for Domain Generalization
- Provable Domain Generalization via Invariant-Feature Subspace Recovery
- Geodesic flow kernel for unsupervised domain adaptation
- Unsupervised Domain Adaptation with Progressive Domain Augmentation
- Grassmannian graph-attentional landmark selection for domain adaptation
- Deep Domain Adaptation by Geodesic Distance Minimization
- VisDA: The Visual Domain Adaptation Challenge
- Self-ensembling for visual domain adaptation