Generating Only the Images That Break the Tie

When a model trained on labeled photographs must work on unlabeled paintings or sketches, the standard fix is to make the two domains' features look statistically alike. That fix is blind to labels. Many different mappings match the distributions equally well, and some of them swap classes, for example sending source "cats" to target "dogs." Identifiability Conditions for Domain Adaptation proves this for linear maps. It also finds that the usual alignment score has essentially no correlation with target accuracy. A second family of methods sidesteps the problem by asking a large text-to-image generator to paint labeled target-style examples. Those pipelines pay for the label information by generating tens of thousands of images, most of them redundant: AttrSyn reports its accuracy plateauing after 15k–20k images.

This note proposes using the generator only where alignment is genuinely ambiguous. First, train a cheap committee of adapted classifiers that all fit the unlabeled data equally well. Second, find the target regions and class pairs on which they disagree. Third, generate a small number of class-conditional, target-styled images for exactly those contested pairs, and use them mainly to decide which committee member is right, not as bulk training data. The bet is that resolving a symmetry needs a handful of correctly labeled anchors in the right place, not a volume of synthetic data spread uniformly over classes.

Core Claim

Unsupervised alignment leaves behind a small, discoverable set of contested decisions. These are target clusters where plausible mappings assign different labels, and they are typically confined to a few class pairs. A generator conditioned on those classes and on the target style can supply the missing label information. Spending generation budget only there should recover most of the accuracy of large untargeted synthesis at a small fraction of the image count.

The novel ingredient is the routing signal: disagreement among equally plausible adapted hypotheses, rather than a single model's uncertainty or diversity in a fixed embedding. The two can come apart. A model that has learned a class-swapping alignment is confidently wrong, so its prediction entropy is low exactly where the error is. Disagreement across the set of plausible maps, in the spirit of the worst-case-over-maps (WOMP) analysis of Gulrajani and Hashimoto, flags the error anyway.

Research Program

All adaptation happens on frozen features from a pretrained image encoder such as CLIP-ViT-L or DINO. This keeps the committee cheap enough to sidestep a known cost. WOMP is expensive because it needs many randomly restarted models, and here each restart is only a small alignment head trained in minutes.

The committee is built in three steps. We train $M$ alignment heads, each a feature-matching objective plus a source classification loss. Each starts from a different random initialization, and some also start from a random orthogonal rotation of the target features. This rotation trick is borrowed from the randomized consistency test that Gulrajani and Hashimoto call CERTIFY. We keep only the heads whose source accuracy and alignment loss fall within a tolerance of the best head.

We then cluster the target features and compute, per cluster, how often the surviving heads assign different labels. The output is a ranked list of contested clusters, each with the small set of classes the heads argue over.

For each contested cluster, we write prompts that combine the contested class names with a description of the target style. The style words come from CLIP retrieval against the cluster centroid, so no large captioning model is needed. A diffusion model renders a small batch of images per contested class.

These anchors are used in two ways. The cheap use is selection: we score each committee member by its accuracy on the anchors, keep or reweight the consistent members, and recompute disagreement. The second use is light fine-tuning of the selected head on the anchors. Generation stops when committee disagreement falls below a threshold or the budget runs out, so total spend tracks how ambiguous the task actually was.

Two analyses make this a study and not only a method. The first asks whether committee disagreement predicts target error better than single-model entropy across benchmarks. The second asks how much of the selection benefit survives generator mistakes. Diffusion models can misrender fine-grained classes or alter object layout, and SIDA acknowledges this drift. We therefore measure how often a human rater judges the anchors correctly labeled, and how accuracy degrades as we deliberately corrupt a fraction of anchor labels. The selection step should tolerate noisier anchors than fine-tuning does: choosing among a few hypotheses needs only a majority of correct anchors, whereas fine-tuning on mislabeled images directly installs the error.

Relation to Prior Work

Three papers motivate the gap. Gulrajani and Hashimoto show that alignment can be underspecified and propose pessimism over plausible maps, but they never inject outside information to resolve it. Co-regularized Alignment trains diverse classifiers and forces them to agree; we instead treat their disagreement as a pointer to where external labels are needed. The poisoning studies of Mehra et al. show how far label-blind alignment can fall, with target accuracy collapsing while alignment metrics look perfect (On the Effectiveness of Poisoning against Unsupervised Domain Adaptation, Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning). Their result says the missing ingredient is target-side semantic anchors, which is what we try to supply cheaply.

Three synthesis papers supply methods and serve as foils. AttrSyn builds attribute-rich prompts for SDXL and trains a linear probe on frozen CLIP features. Its generation is driven by language-model priors with no look at the unlabeled target, and it saturates. SIDA style-transfers source-like synthetic images into target conditions with substantial upfront cost. DUQGen is the closest in spirit: it cuts synthetic retrieval queries from about a million to 1,000 by choosing diverse documents via clustering in an unadapted encoder. Its selection, however, is purely geometric and never asks whether the chosen points resolve label ambiguity. Its own ablation shows that removing clustering drops performance below the zero-shot baseline, which shows how much such pipelines depend on the selection rule.

Partial Identifiability for Domain Adaptation constrains the design. It obtains identifiability from structural assumptions across multiple source domains. We do not assume that structure, and in the single-source setting we expect structural constraints alone to leave residual ambiguity that anchors must resolve.

Baselines

The naive reference points are the source-only linear probe on frozen features and zero-shot CLIP with class-name prompts. Both use no target information beyond the encoder.

The strongest existing method on our main fine-grained benchmark is AttrSyn. Its reported result on CUB-200-Painting with a CLIP-ViT-L backbone is 56.51% accuracy, a reported 13.62% improvement over zero-shot CLIP. We will rerun AttrSyn with the same diffusion model, encoder, and linear-probe protocol at several image budgets, from a few hundred up to its 15k–20k plateau. Beating it means one of two things: matching 56.51% (or our reproduced number) with at least five times fewer generated images, or clearly exceeding it at equal budget. We will also report wall-clock generation cost, since the claim is about overhead.

For classical unsupervised adaptation without generation, we run Co-DA's agreement objective on the same frozen features. On PACS we report the published reference points while noting that their settings differ from ours. Those points are TAST test-time adaptation at 81.94% with ResNet-18, and iMSDA at 93.48% average in the multi-source setting. On Office-Home, iMSDA reports 76.19%.

As an oracle-side reference, we run Discrepancy-Based Active Learning with real target labels, using its weighted K-medoids selection at the same number of labeled points as our anchors. This tells us how close generated anchors get to true labels.

Three ablations isolate the routing signal. All share the identical generator, prompt construction, budget, and stopping rule; only the choice of where to generate changes. The first allocates the same budget uniformly across classes. The second routes generation by single-model prediction entropy. The third routes by DUQGen-style cluster diversity in the unadapted encoder. A fourth ablation compares selection-only use of anchors against fine-tuning-only use. All methods get the same hyperparameter search budget. Hyperparameters are tuned on a held-out source-domain split, with no target labels, so that we do not quietly give any method target supervision.

Why not just…?

Why not just generate a balanced synthetic set for every class, as AttrSyn does? This is the main threat. With a strong encoder, uniform generation may already cover the contested classes well enough. The uniform-allocation ablation at matched image counts settles it. If targeted routing shows no advantage at small budgets, where allocation matters most, the core claim fails. We expect the gap to be largest when contested classes are few relative to the total class count, as in fine-grained CUB, and to vanish on easy shifts.

Why not just route generation by one model's uncertainty? Entropy is free and needs no committee. But a class-swapping alignment is confident, so entropy is low on exactly the points it gets wrong. We test this two ways. On the benchmarks, we compare how well entropy and disagreement predict error. In a controlled setting with known symmetry, we take Colored MNIST, where Gulrajani and Hashimoto document the miscalibration WOMP corrects, and add an injected label-swapping symmetry. If entropy routing performs equally well on natural benchmarks, the honest conclusion is that real shifts rarely produce confident swaps, and the committee adds only cost.

Why not just use zero-shot CLIP text prompts as the anchor, with no image generation at all? Text anchors are cheaper than diffusion and carry the same class semantics. They may suffice whenever the encoder's text and image spaces agree on the target style. We include this as an alternative oracle within the same routing pipeline. The committee scores against text-embedding predictions instead of generated images. If it matches generated anchors, the paper's conclusion becomes that routing matters and generation does not. That would be a useful negative result about synthesis overhead. We expect images to win on styles like sketches and paintings, where text-image agreement in CLIP is weakest, but we do not assume it.

Why not just label a few target images by hand? When a human annotator is available, that is the better choice, and the active-learning reference measures how far generated anchors fall short. The case for generated anchors is restricted to settings where target labels are slow, expensive, or unavailable.

Proposed Experiments

The main benchmarks are CUB-200-Painting (photos as source, paintings as target), PACS with each single domain as source, and Office-Home's four domains. We also use a controlled Colored MNIST or rotated-cluster task where the ambiguity is planted and known.

The primary metrics are target accuracy as a function of generated images and of generation GPU-hours, plus the area under that curve. Secondary metrics are expected calibration error, and how well per-cluster disagreement predicts per-cluster error, measured by rank correlation.

The decisive checks are three, all at budgets of at most 1–2k images on CUB-200-Painting. First, disagreement-routed generation should beat uniform and entropy routing at the same image count. Second, selection using anchors should recover most of the gain even when a quarter of the anchor labels are corrupted. Third, the stopping rule should spend little on easy shifts and more on hard ones, demonstrating that budget adapts to measured ambiguity.

If the first check fails, the routing idea is wrong and we report that. If only the text-anchor variant survives, we report that generation was unnecessary overhead for this class of problems.