Retrieving Real Target Patches to Guide Unpaired Translation for Domain Adaptation

When a driving-scene segmenter trained on clear-weather images is deployed in rain, its accuracy drops sharply. A common fix is to translate the labeled clear images into rainy-looking ones and train on them. Unpaired image-to-image translation does this: it learns to map images from one domain to another without matched pairs. In practice, these translators often change only the global color tone. They miss the local evidence of rain, such as droplets, glare, and wet-road reflections. The reason is that a discriminator, the network that judges whether an output looks like a real target image, only checks whether outputs look realistic overall. Nothing ties a particular output patch to a particular kind of real target patch. Machine translation faced a similar problem when adapting to new domains with only monolingual text. One effective answer there was retrieval rather than more generation. kNN-MT, short for k-nearest-neighbor machine translation, looks up concrete target-domain examples in a datastore and lets them shape the output.

We propose to apply this idea to images. During training, each output patch of the translator is pulled toward its nearest real target-domain patches of the same semantic class. These patches are retrieved from a datastore built from unlabeled target images. The datastore keys depend on the segmenter's own pseudo-labels on target images, and those pseudo-labels improve as the segmenter improves. Translation, retrieval and self-training therefore refine one another, much as forward and backward models refine each other's datastores in iterative kNN-MT. Our central claim is testable: class-conditioned patch retrieval supplies the local target appearance that, according to Domain Bridge for Unpaired Image-to-Image Translation and Unsupervised Domain Adaptation, required an extra web-crawled video "bridge" domain. If the claim holds, the retrieval version should match that pipeline on clear→rainy segmentation without using any bridge data.

Core Claim and Positioning

The method borrows from three lines of work, each in a different role.

The machine-translation side supplies the mechanism. Exploiting Target Language Data for Neural Machine Translation Beyond Back Translation builds pseudo-pairs by retrieving target-domain sentences through cross-lingual embeddings. It reports an average gain of +4.51 BLEU without regenerating data. Iterative Nearest Neighbour Machine Translation for Unsupervised Domain Adaptation shows that letting two models iteratively refresh each other's datastores beats a static datastore by about 1 BLEU on average. We reuse both ideas: retrieval in place of synthesis, and iterative refresh of the retrieval keys.

Both NMT papers also warn that querying a datastore adds latency at inference time. That objection mostly disappears in our setting. Retrieval happens only while training the translator, and only the segmenter is deployed.

The image side supplies the base translator and the evidence for the gap. Contrastive Learning for Unpaired Image-to-Image Translation introduced CUT (Contrastive Unpaired Translation). CUT keeps content by matching corresponding input and output patches through a contrastive loss, and draws its "negatives" (patches that should not match) from inside the same input image. It drops the round-trip cycle constraint, which forces bijective mappings and invites hidden steganographic artifacts. CUT's contrastive term acts only between input and output, so it preserves content but says nothing about which target appearance a patch should take on.

Synthetic-to-Real Domain Adaptation using Contrastive Unpaired Translation shows that training CUT on small patches suppresses geometric hallucination. On YCB-Video it raises segmentation from 0.701 to 0.763 mIoU (mean intersection over union), close to the 0.770 obtained with real labels. Domain Bridge shows that plain translation misses local rain traits. It recovers them with bridge videos, multimodal style sampling and weighted pseudo-labels, moving BDD-rainy from 31.67 to 40.04 mIoU.

Image to Image Translation for Domain Adaptation acts as a constraint on our design. Its multi-network cycle-consistent system reaches 35.7 mIoU on GTA5→Cityscapes, up from 21.1 for the source-only model. However, it could not run full cycle consistency at high resolution because of memory limits. Our design therefore stays one-sided and keeps the datastore off the generator's backward pass.

What is new is the target-side retrieval loss and its coupling to self-training. No paper in the file queries a datastore of real target image patches to guide translation. The closest work, Domain Bridge, gets local target appearance from additional paired-in-time data rather than from the unlabeled target set itself.

Research Program

We start from a patch-trained CUT translator $G$ that maps clear images to rainy ones. A frozen pretrained encoder $\phi$ computes patch features.

The datastore holds features $\phi(t_q)$ of patches $t_q$ cropped from unlabeled target images. Each patch is keyed by two things: the segmenter's pseudo-label for that patch, and a content descriptor. The content descriptor is the feature of a weather-suppressed version of the patch, obtained through a grayscale, low-frequency projection.

For each output patch $p$ of $G(x)$, we know its class $c$ from the source label of the clear image $x$. We retrieve its $k$ nearest same-class target patches $\mathcal{N}_c(p)$ by content descriptor. We then add the following retrieval-contrastive term to CUT's objective:

$$ \mathcal{L}_{\text{ret}} = -\log \frac{\sum_{r \in \mathcal{N}_c(p)} \exp(\phi(p)^\top \phi(r)/\tau)}{\sum_{r \in \mathcal{N}_c(p)} \exp(\phi(p)^\top \phi(r)/\tau) + \sum_{n \in \mathcal{M}_{\bar c}} \exp(\phi(p)^\top \phi(n)/\tau)} $$

Here $\mathcal{M}_{\bar c}$ is a set of retrieved target patches from other classes, and $\tau$ is a temperature. In plain terms, a translated road patch should look like the real wet-road patches it most resembles, and unlike real sky or car patches.

Every few epochs, we retrain the segmenter on translated images plus confidence-weighted target pseudo-labels, using the same weighting rule as Domain Bridge's pseudo-labeling. We then re-key the datastore with the updated pseudo-labels. We will measure how pseudo-label accuracy and segmentation mIoU change across refresh rounds. This tests whether the refinement loop actually helps or instead amplifies early labeling errors, which is the failure mode known from back-translation.

We will also report datastore size, retrieval cost per training step and total GPU memory against CUT and CycleGAN. A training-time memory penalty would weaken the case against the multi-network system of Image to Image Translation for Domain Adaptation.

Baselines

The naive reference point is a segmenter trained only on clear Cityscapes images and evaluated on BDD-rainy. Domain Bridge reports 31.67 mIoU for this setting, and we will reproduce it with our segmenter. On GTA5→Cityscapes, the corresponding reference is 21.1 mIoU, as reported by Image to Image Translation for Domain Adaptation.

The strongest existing method on our main question is the full Domain Bridge pipeline: bridge domain, online multimodal style sampling and weighted pseudo-labels, reported at 40.04 mIoU on BDD-rainy. It is the right competitor because it explicitly targets the local-rain-trait failure we claim to fix. It also uses extra bridge video data, which our method does not. To keep the comparison fair, we will run our method with the same segmentation network, the same clear and rainy image sets, the same pseudo-label weighting rule and the same training budget in GPU-hours. If our reproduction of Domain Bridge differs from the published 40.04, we will report both numbers. Beating Domain Bridge means matching its mIoU within run-to-run variance (three seeds) without bridge data, or exceeding it by more than one standard deviation.

The translation baselines are CycleGAN, full-image CUT and patch-trained CUT. CUT outperforms CycleGAN on Cityscapes label→photo, with 24.7 versus 20.4 mAP. On Horse→Zebra its FID is 45.5 versus 77.2, where FID (Fréchet Inception Distance) measures distance from real images and lower is better. Each translator is followed by the same self-training stage, so any difference comes from translation alone.

The key ablations isolate retrieval. The first replaces nearest-neighbor retrieval with random same-class target patches. This tests whether matching content matters, or whether class-level target statistics are enough. The second removes class conditioning. The third freezes the datastore keys after the first round, which removes the refinement loop.

Why not just…?

Why not just train CUT on small patches, which already focuses it on local texture? This is the most serious threat. The patch-based study showed that local training captures sensor noise and texture on YCB-Video. No result in the file tests it under the clear→rainy gap, where Domain Bridge reports that translators flatten rain into color shifts. Patch-trained CUT is therefore a direct baseline in our study. If it reaches the same BDD-rainy mIoU as our method, the retrieval loss adds nothing, and we will report that outcome as the finding.

Why not just paste retrieved real target patches onto source images instead of generating anything? This is the non-generative alternative raised by the NMT pseudo-datastore results, and it is a plausible competitor. We will run class-matched patch compositing with blending at boundaries, using the same datastore. We expect seams and mismatched geometry, such as reflections that do not follow the road, to hurt the segmenter. That expectation is unverified, so this comparison decides whether generation is needed at all.

Why not just self-train on target pseudo-labels without any translation? Domain Bridge's weighted pseudo-labels are part of its pipeline, but the file gives no number for self-training alone on BDD-rainy. We will run it under the same schedule. If it closes most of the gap from 31.67, translation itself is the questionable component, regardless of how it is guided.

Proposed Experiments

The main experiment is Cityscapes→BDD-rainy semantic segmentation, measured in mIoU over three seeds against all baselines above. The second setting is GTA5→Cityscapes, where the reference result is 35.7 mIoU from Image to Image Translation for Domain Adaptation. It checks that retrieval helps outside weather shift and at high resolution. The third is YCB-Video, which has only 0.007 mIoU of headroom below the real-label upper bound. It serves as a sanity check that retrieval does not hurt where patch CUT already nearly succeeds.

We will also measure Horse→Zebra FID with class conditioning removed. This checks that the loss does not reduce image quality relative to CUT's 45.5. For the Cat→Dog benchmark, no reported result was found on Lacuna, so it is optional.

The decisive checks are four. The method must beat patch-trained CUT on BDD-rainy under equal compute. It must match Domain Bridge without bridge data. Nearest-neighbor retrieval must outperform random same-class retrieval. Per-class gains should concentrate in road, sidewalk and vehicle classes, where reflections and droplets appear.