Many Anchors, One Simulator: Keeping Ensembles Diverse Through Synthetic Fine-Tuning

Ensembles help only when their members make different mistakes. Sim-to-real training may be the worst case for that assumption. Every member learns the task from the same simulator, so every member is pushed toward the same simulator-specific shortcuts: rendering lighting, clean CAD textures, idealized occlusion. Meanwhile, the real-world knowledge each member brought from pretraining is steadily overwritten. We hypothesize that synthetic fine-tuning is a strong homogenizer. Even ensemble members that start diverse should drift toward shared failures on real images, and seed-based or same-checkpoint ensembles should begin as near-clones.

We propose to measure this effect directly and then counter it. Each ensemble member is tied during synthetic fine-tuning to a different real-image pretrained model, which we call its anchor. The simulator supplies the shared task signal. Each anchor supplies a distinct prior about what real images look like, and that prior is what keeps members' real-world errors apart. The question is whether spending regularization this way yields real-domain ensembles that are both accurate and genuinely independent. The comparison points are anchoring every member to the same ImageNet model, or not anchoring at all.

Core Claim

Automated Synthetic-to-Real Generalization (ASG) showed that the main enemy in sim-to-real transfer is forgetting. ASG fine-tunes a single ImageNet model on synthetic data while penalizing its divergence from a frozen ImageNet copy. This "proxy guidance" alone lifts VisDA-17 real accuracy from 28.2% to 58.7%, and full ASG reaches 61.1%.

What is being transferred in transfer learning? showed that models fine-tuned from the same pretrained weights stay in one loss basin and behave as functional twins. Together these results imply a tension. Building an ensemble of ASG-style models, all anchored to one checkpoint, should produce members that are well generalized but redundant.

Our claim has two parts. The first is empirical: on real target data, pairwise error agreement among ensemble members rises over the course of synthetic fine-tuning. For already-diverse members, this rise should track the same forgetting curve ASG documents. The second is methodological: giving each member its own real-image anchor, rather than one shared anchor, preserves between-member independence on real data, and does so without giving up the single-model gains that proxy guidance delivers.

What is new is treating the choice of anchor as the source of ensemble diversity in sim-to-real transfer, together with a quantitative account of how the simulator erodes diversity.

Research Program

Measurement study. We first measure how diversity decays. Ensembles of five members, all with a fixed ResNet-50 architecture, are trained on VisDA-2017 synthetic CAD renderings (VisDA: The Visual Domain Adaptation Challenge) under three conditions:

  • same ImageNet initialization with different seeds;
  • different real-image pretrained initializations, meaning public ResNet-50 checkpoints trained with different objectives or on different real corpora;
  • the second condition plus ASG-style single-anchor guidance.

At regular checkpoints we evaluate every pair of members on the labeled real VisDA validation split. We use the labels only for measurement, never for training. The three diversity measures are:

We also report the fraction of real images in a "misleading tier." Following Hidden Clones, this is the set of images where at least one member is right but the vote is wrong. Hidden Clones measured this tier only in-domain. Here we test whether it grows under sim-to-real shift.

Method. The method changes one thing relative to ASG. Member $k$, with backbone features $f_k$, is trained with

$$\mathcal{L}_k = \mathcal{L}_{\text{task}}(x_s, y_s) + \lambda\, \big\| P_k f_k(x_s) - a_k(x_s) \big\|^2 .$$

In this objective, $(x_s, y_s)$ are labeled synthetic images, $a_k$ is the frozen penultimate-layer feature map of member $k$'s own anchor, $P_k$ is a learned linear projection, and $\lambda$ sets the anchor strength. We use feature matching rather than ASG's output-level divergence because self-supervised anchors have no classifier head. Adding a projection is a real change from ASG, so we also run a single-anchor feature-matching variant. That variant separates the effect of the loss form from the effect of anchor diversity.

Each member is initialized from its own anchor. The anchors are chosen to be as unlike each other as possible, measured by their real-image representation similarity on a small held-out set of real images. That set is unlabeled from the method's point of view.

Extension to adaptation and dense prediction. The main setting is pure synthetic-to-real generalization, with no target data seen during training. This is the ASG setting and the cleanest test of the idea. We then check two things. The first is whether anchored ensembles make better teachers for unsupervised domain adaptation, meaning adaptation that uses unlabeled target images. The second is whether the result transfers to GTA5-to-Cityscapes semantic segmentation, where "same wrong class" is computed per pixel.

Baselines

Naive reference. A single ImageNet-initialized model fine-tuned on synthetic data only. ASG reports 28.2% on VisDA-17 and 22.17% mIoU for FCN-Res50 on GTA5-to-Cityscapes. Any ensemble must clear this easily; the naive model only sets the scale.

Strongest single-model competitor. ASG itself. The file reports 61.1% VisDA-17 accuracy and 29.64% mIoU on GTA5-to-Cityscapes. We reimplement it on our backbone rather than copying numbers, because the file does not state ASG's VisDA backbone. ASG is the method most likely to make ensembling look unnecessary.

Strongest ensemble competitor, and the key ablation. An ensemble of five members all trained with ASG proxy guidance toward the same ImageNet anchor, with different seeds. It isolates our key ingredient: the only difference from our method is whether anchors differ. A close second is an ensemble of the same diverse pretrained initializations fine-tuned without any anchoring. This separates "diverse starting points" from "diverse starting points held in place."

All ensembles use five ResNet-50 members, identical synthetic data and augmentation, equal training epochs, and the same aggregation (probability averaging, plus the hierarchical family voting of Hidden Clones as a sensitivity check). The hyperparameter $\lambda$ gets the same search budget in every condition. Model selection uses a held-out synthetic validation split only. Selecting on real labels would leak target information, which is the evaluation hazard Synthetic Data, Real Errors warns about.

Beating the strongest baseline means two things. The anchored ensemble must reach higher real-target accuracy than both the same-anchor ASG ensemble and the unanchored diverse ensemble, with the gain holding across three repeated runs. It must also show lower conditional error agreement and a smaller misleading tier than both.

Adaptation reference. For the adaptation extension, the reference is IMED (Instance-aware Model Ensemble With Distillation). IMED fuses ensemble members with per-image weights predicted by a small auxiliary network, then distills them into one student; it reports 89.9% on VisDA-2017. IMED never examines where its members' diversity comes from. We therefore swap our anchored members into its teacher at fixed compute and test whether its per-image fusion improves. We do not claim to beat 89.9% outright.

Why not just…?

Why not just ensemble different pretrained backbones and fine-tune them normally? This is the most serious threat. If diversity in pretraining survives synthetic fine-tuning on its own, anchoring adds nothing. Our prediction rests on ASG's finding that fine-tuning curves peak early and then collapse toward simulator artifacts, which implies that diverse members forget toward the same place. The unanchored diverse ensemble is run as a primary baseline for exactly this reason. If its error agreement stays flat over training, we will report that the practical lesson is "diversify initializations," and that the measurement study is the contribution.

Why not just train seed ensembles, perhaps on domain-randomized data? Synthetic Data, Real Errors states that seed-level ensembles cannot correct systemic biases shared by the data source. Neyshabur et al. show that seeds do not escape the shared pretrained basin. Domain randomization changes rendering parameters but still exposes every member to the same randomized distribution. Our same-checkpoint seed condition directly measures how redundant these members are on real images.

Why not force members to disagree on unlabeled real images? Two lines of prior work show that disagreement objectives can build diversity. DiffDiv makes members disagree on diffusion-generated counterfactuals, and UDEED makes them disagree on unlabeled in-distribution data. On real target images, though, forced disagreement has no label to check it and can push members onto wrong answers. It also requires target data, which the generalization setting forbids. In the adaptation extension we run a UDEED-style real-target disagreement term as an additional baseline, and also combined with anchoring, to see whether the two ingredients complement each other.

Why not just use one larger model? Kim et al. find that more accurate models make more correlated errors, not fewer. In VisDA terms, a single model has no voters to diversify. A single model with five times the parameters of one member is a worthwhile compute-matched reference. We include it where feasible.

Decisive Checks

The first check is the decay curve. For the unanchored diverse ensemble, conditional error agreement on real VisDA images should increase over synthetic fine-tuning. For the same-anchor ASG ensemble, agreement should start high and stay high. If neither pattern appears, the premise fails.

The second check is the matched comparison. At equal members, epochs, and tuning budget, the diverse-anchor ensemble should beat the same-anchor ensemble on real accuracy. It should also show fewer effective-voter losses and a smaller misleading tier, on both VisDA-C and GTA5-to-Cityscapes.

The third check is dose-response. As $\lambda$ grows, single-member real accuracy should saturate, as in ASG, while ensemble gains keep tracking how different the anchors are. We test this by varying anchor dissimilarity at a fixed $\lambda$. If ensemble accuracy does not rise with anchor dissimilarity, the mechanism we propose is not what drives the result.