When a model is trained on simulated or game-engine data and adapted to real images that have no labels, there is no honest way to choose which training checkpoint to ship. Checking accuracy on the target domain would require exactly the real-world labels the method is meant to avoid. Practitioners therefore ship the last checkpoint, or quietly early-stop using target labels that a deployed system would not have. Many adaptation methods are adversarial. A feature or segmentation network tries to fool a "domain discriminator" that guesses whether an input came from simulation or reality. Training is a two-player game, and optimization theory says such games tend to orbit their solution rather than settle on it. We therefore expect the final checkpoint of adversarial sim-to-real training to be unreliable, and more unreliable than in ordinary supervised learning, where a labeled validation set can hide the problem.
This proposal treats last-iterate convergence, meaning guarantees about the final parameters rather than about an average over the trajectory, as the missing ingredient for label-free model selection. We make two claims. First, optimizers built for last-iterate stability in games should shrink the gap between the final checkpoint and the best checkpoint chosen in hindsight with target labels. Second, the quantity those theories drive to zero, the norm of the game's joint gradient field, can be estimated without target labels and used as a stopping certificate. The closest prior work either proves these rates on toy games and CIFAR-10 image generation, or runs sim-to-real adversarial adaptation with plain alternating Adam or SGD. No work in the file measures or targets the final-versus-best-checkpoint gap in adaptation.
Core Claim
In an adversarial adaptation game, collect the segmentation or classification parameters $u$ and the discriminator parameters $v$ into $z=(u,v)$. Write the game's vector field as $F(z) = (\nabla_u \mathcal{L}, -\nabla_v \mathcal{L})$, where $\mathcal{L}$ is the task loss on labeled simulated data plus the weighted adversarial loss on unlabeled real data. Plain simultaneous or alternating gradient steps follow the rotational part of $F$ and circle the equilibrium. Averages of such orbits can sit at the equilibrium even while the final point never gets there, as Hsieh et al. and Loizou et al. discuss.
We claim three things. First, in the standard sim-to-real games, the output-space adversarial segmentation of AdaptSegNet and the pixel-level generator game of PixelDA, the selection gap is large and varies from seed to seed. We define the selection gap as target accuracy of the best-in-hindsight checkpoint minus target accuracy of the final checkpoint. Second, replacing the optimizer with a last-iterate method shrinks this gap at matched gradient budget. Third, a held-out, label-free estimate of $\|F(z_t)\|^2$ tracks late-training target accuracy well enough to serve as a stopping and selection signal, better than source validation accuracy or discriminator loss.
The contribution is empirical and methodological. We are not claiming a new convergence theorem for deep networks. Extending monotonicity-based guarantees to deep non-monotone networks is an open problem, and we do not pretend to solve it.
Research Program
We borrow three damping mechanisms from the last-iterate literature and adapt each to the adaptation setting.
The first is linear interpolation. The Lookahead-style Krasnosel'skiĭ–Mann step of Pethick et al. runs a few fast optimizer steps and then moves the slow weights halfway toward the fast weights. It resets the fast weights to the slow ones, using the interpolation weight 0.5 that their analysis identifies as best. That paper supplies the method and the strongest prior evidence: on CIFAR-10 image generation, Lookahead-GDA reached FID 16.87 against 19.36 for plain GDA, and plain Adam diverged. The paper never touched domain adaptation.
The second is anchoring. The extra-anchored gradient method of Yoon and Ryu combines an extragradient look-ahead step with a pull toward an anchor point, and that pull decays over training. It achieves the optimal $O(1/k^2)$ rate on $\|F(z_k)\|^2$ for smooth convex-concave problems. In adaptation there is a natural anchor that toy games lack: the source-only model trained on simulated data alone. Anchoring the segmentation network to it resembles how DAFormer tethers features to ImageNet weights. The difference is that the anchor pull decays on the Halpern schedule rather than staying as a fixed penalty.
The third is single-call optimism. The past-extragradient and optimistic updates unified by Hsieh et al. reuse the previous gradient to extrapolate. They cost one backward pass per step instead of two, which matters for ResNet-101 segmentation.
We also build the label-free certificate. At checkpoints we estimate $\|F(z_t)\|^2$ on fixed held-out minibatches. These contain simulated images with labels and real images without labels, so no target labels are needed. The estimate uses one gradient evaluation per player.
For interpolation methods, the natural certificate is the fixed-point residual between slow and fast weights. In the cohypomonotone setting of Pethick et al., this residual is non-increasing across outer loops. We will state and check that property as the formal justification for the stopping rule. To see whether its assumption is remotely plausible in these games, we will locally estimate the cohypomonotonicity constant by finite differences along the trajectory. We will report where the assumption holds and where it fails.
Baselines
The naive reference point is each method's published recipe with its own optimizer, deploying the final checkpoint. Concretely, that means AdaptSegNet with a DeepLab-v2 ResNet-101 on GTA5 to Cityscapes, and PixelDA on MNIST to MNIST-M and synthetic to real LineMod. The strongest reported numbers for these setups are 42.4 mIoU for multi-level AdaptSegNet (AdaptSegNet), 98.2% accuracy on MNIST-M and 23.5° mean angle error on LineMod (PixelDA). The file does not say whether these figures come from final or selected checkpoints. That ambiguity is part of what we measure: we re-run each recipe across at least five seeds and report both.
The non-deployable upper reference is the best-in-hindsight checkpoint selected with target labels. It defines the selection gap.
The strongest competitor is an exponential moving average of weights applied to the same Adam trajectory. For a purely rotational orbit, an average lands near the center. Averaging is therefore the simplest thing that could erase the whole effect, and it is what mean-teacher self-training already does.
The second strong competitor is a well-tuned annealed learning rate. It decays to zero, as analyzed by Attia and Koren, or switches from constant to decaying in the manner of Loizou et al.. Annealing shrinks the noise-driven wandering of the final iterate without changing the update direction.
For VisDA-C classification we use DANN, reported at 57.4% mean accuracy, and CLS, reported at 81.6%, both from CLS. CLS is the strongest adversarial method in the file on GTA5 to Cityscapes, at 53.0 mIoU. We will apply our optimizer to its adversarial component if the joint classifier-discriminator can be isolated cleanly. DAFormer, at 68.3 mIoU, serves as an external reference for what non-adversarial self-training reaches with a transformer. It is not a like-for-like competitor.
Fairness rules apply to every arm. All arms share the backbone, data, augmentation and total number of gradient evaluations. EAG gets half as many steps because each step costs two passes. Every arm gets the same hyperparameter search, twenty configurations tuned only with source-validation and label-free criteria, so no method benefits from target-label tuning.
Two ablations isolate the key ingredient. The first replaces the decaying Halpern anchor with a constant L2 penalty toward the source-only weights, which asks whether the anchoring schedule matters or plain regularization suffices. The second applies Lookahead to the segmentation network only, leaving the discriminator on Adam, which asks whether damping the game matters or only damping one player.
A method beats the field if it does three things. It must cut the median selection gap by at least half relative to both EMA and annealing. It must reduce seed-to-seed standard deviation of final-iterate accuracy. It must not lower best-in-hindsight accuracy. Merely raising the best checkpoint does not count.
Why not just…?
Why not just keep an exponential moving average of the weights? This is the real threat, and we treat it as a possible outcome rather than a strawman. Averaging recovers the center of a clean orbit. It can fail when the trajectory drifts between distinct basins, or when batch-normalization statistics and the discriminator no longer match the averaged network. The file's claims about averaging harming non-convex models are general arguments, not measurements in adaptation. The head-to-head comparison at matched budget settles it. We will also report the certificate at the averaged point. If EMA closes the selection gap as well as Lookahead does, the conclusion is that sim-to-real instability is mainly rotation around a good point, and averaging is enough. That would still be a useful, publishable answer.
Why not just anneal the learning rate or stop on source validation accuracy? Annealing controls noise, but it does not remove the rotational component of the game, so a shrinking step can freeze the iterate at an arbitrary phase of its orbit. Source accuracy measures fit to simulation, which is exactly the thing that fails to transfer. Both are included as arms. We will also compute the rank correlation between each label-free signal and target accuracy across late checkpoints. The signals are source accuracy, discriminator loss, prediction entropy and our residual. If annealing plus source validation matches us, the proposal's optimizer claim fails and only the certificate question remains.
Why not just drop adversarial training and use DAFormer-style self-training? For standard driving benchmarks with ImageNet-pretrained transformers, this may well be the practical answer. We do not claim adversarial methods will beat 68.3 mIoU. Our claim is narrower. Wherever an adversarial game is used, as in pixel-level translation, pose estimation or sensors without large pretrained models, its final checkpoint should be trustworthy. Self-training has its own coupled loop: a teacher labels data and the student learns from it. Mean-teacher updates amount to Lookahead without resetting the fast weights. In a secondary experiment on the DAFormer codebase, we test whether adding that reset changes the pseudo-label flip rate or the selection gap. The flip rate is the fraction of target pixels whose pseudo-label changes between consecutive teacher updates.
Proposed Experiments
The decisive experiment runs AdaptSegNet on GTA5 to Cityscapes with five optimizer arms: Adam, EMA, annealed learning rate, Lookahead, and anchored extragradient. Each arm uses at least five seeds, with target mIoU logged every 1% of training. We report final mIoU, best-in-hindsight mIoU, selection gap, spread over the last 20% of checkpoints, and seed standard deviation.
The same protocol is repeated on MNIST to MNIST-M and synthetic to real LineMod with PixelDA. There, generator-discriminator rotation should be strongest. We also run it on VisDA-C with DANN.
The file reports no selection-gap or seed-variance figures for any of these methods. The file has no reported result on Lacuna for final-versus-best checkpoint gaps in adversarial UDA, so the Adam baseline numbers are our own first measurement.
The second decisive check concerns the certificate. We ask whether stopping when the residual plateaus yields target accuracy within one point of the best-in-hindsight checkpoint, and whether it does so more often than the other label-free criteria.
A negative result on the first check falsifies the optimizer claim: no damping method shrinking the gap beyond EMA and annealing. A negative result on the second falsifies the certificate claim: a weak correlation between the residual and target accuracy. Both are informative for anyone deploying adapted models without target labels.
References
- On the convergence of single-call stochastic extra-gradient methods
- Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity
- Learning to Adapt Structured Output Space for Semantic Segmentation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Stable Nonconvex-Nonconcave Training via Linear Interpolation
- Accelerated Algorithms for Smooth Convex-Concave Minimax Problems with O(1/k^2) Rate on Squared Gradient Norm
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
- Benefits of Learning Rate Annealing for Tuning-Robustness in Stochastic Optimization
- Adversarial Unsupervised Domain Adaptation with Conditional and Label Shift: Infer, Align and Iterate