In unpaired image translation, a model learns to turn horses into zebras or summer scenes into winter ones from two separate photo collections, with no matched before/after pairs. These models often change the wrong things. Zebra stripes bleed onto grass, and colors shift in regions that should stay the same. Language modeling faced a similar problem when adapting models to specialized text. Hiding random words for the model to predict wastes training on words like "the." Hiding the words that make a domain distinctive, such as "inhibitor" in chemistry papers or "excellent" in reviews, adapts the model faster and better. We propose bringing that idea into translation. First, build an unsupervised "visual vocabulary" of image patches from both collections. Second, score each visual word by how much more often it occurs in the target collection than in the source, as text methods score domain keywords. Third, train the generator to fill in masked target-domain keywords from the surrounding context.
The practical effect is to recast part of translation as a self-supervised infilling task on real target images, where ground truth exists. The generator learns to redraw a zebra's coat from a blurred, grayscale silhouette plus the untouched grass around it. At test time, translating a horse becomes the same infilling problem: the horse's divergent patches are infilled while shared patches are left alone. The two-sentence version is this: score image regions by how specific they are to each domain, then teach the translator by masking and reconstructing only the target-specific regions. Doing so should concentrate the generator's learning on what must change and keep it from altering what should not.
Core Claim
Our central claim concerns the identity term in contrastive translation, the extra loss that asks the generator to leave a target image unchanged, $G(y)\approx y$. Replacing it with a masked identity objective improves translation quality and background preservation over current contrastive methods, provided the masks are biased toward domain-divergent patches scored by cross-domain log-odds. The gain must come from where the mask is placed, not from masking itself. Uniform masking at the same or higher rate should not reproduce it.
That last condition is the real test. Uniform Masking Prevails in Vision-Language Pretraining found that once masking rates rise to 60–75%, clever selective masking loses its edge over random masking, with uniform masking winning on VQA with ~83% empirical probability at a 60% rate. If that result transfers to translation, our idea fails, and the experiments are built to show whether it does.
Relation to Prior Work
Three text methods supply the scoring machinery, and each relies on a different statistic. Difference-Masking scores words with TF-ICF, a measure of how common a word is in the domain relative to a general corpus. It then spreads masking probability to words with similar embeddings, and it reaches 74.04% vs. 63.74% for random masking on ACL-ARC citation intent. Knowledge Enhanced Masked Language Model for Stance Detection uses a weighted log-odds ratio with an informative Dirichlet prior to find polarizing words. NB-MLM uses Naive Bayes class relevance and reports a ~5x reduction in steps on IMDB.
We reuse the log-odds statistic and the neighborhood spreading. Our "general corpus" is the other image collection, and our class labels are domain membership, which every unpaired translation dataset already provides. The method therefore needs no supervision beyond what translation already assumes. That addresses the supervision-reliance limitation of NB-MLM.
Difference-Masking also tried visual masking, masking "person" boxes in TVQA and improving to 81.73% from 73.75%. Its visual anchor was chosen by hand and served discriminative question answering, not generation. Our anchors are computed from data, and the target task is generative.
On the translation side, four methods motivate and constrain the design. CUT supplies the base: an adversarial loss plus a patch contrastive loss (PatchNCE) that matches each output patch to its input location, with random internal negatives. Its random patch sampling is the thing we change. BFCUT showed that this randomness causes texture bleed, cutting background masked-LPIPS from 50.5% to 34.6% with heat maps from an ImageNet ResNet-50. It also warned that FID can reward bleed, which shapes our metrics. BFCUT's heat maps are generic objectness, not divergence between the specific pair of domains, and they select contrastive queries rather than drive any reconstruction. PUT ranks and prunes negatives by feature similarity and holds the best Horse→Zebra FID in the file (33.82). It never asks whether a patch is domain-specific. AttentionGAN learns soft foreground masks end to end, but they fail on occlusions such as fences, because nothing tells the network what is domain-relevant.
Contrast-GAN needs ground-truth object masks at both training and inference. In none of these four methods is the mask a target for self-supervised reconstruction, and that missing piece is what we add.
Research Program
Building the visual vocabulary. We extract patch features from both collections with a frozen self-supervised image encoder and cluster them into $K$ visual words with k-means, so each patch is assigned one word. For word $w$, let $c^Y_w$ and $c^X_w$ be its counts in the target and source collections and $\alpha_w$ a prior count proportional to its pooled frequency. The anchor score is the prior-smoothed log-odds z-score, following the stance-detection recipe:
$$ s_w=\frac{\log\frac{c^Y_w+\alpha_w}{n^Y-c^Y_w}-\log\frac{c^X_w+\alpha_w}{n^X-c^X_w}}{\sqrt{1/(c^Y_w+\alpha_w)+1/(c^X_w+\alpha_w)}}, $$
where $n^Y$ and $n^X$ are the total patch counts per domain. A large positive $s_w$ marks a target keyword (stripe texture). A score near zero marks shared content (grass, sky). A large negative score marks a source keyword (brown horse coat).
Spreading scores to neighbors. Following Difference-Masking, each patch's score is spread to patches with similar features in the same image. This marks the whole zebra and not just its most distinctive stripes. It is the visual analog of whole-word masking, and it targets the open problem of defining "words" in continuous image features.
Masked identity training. For a target image $y$ we sample a binary patch mask $m$ in which each patch is chosen with probability rising in its spread score, at overall rate $r$. We then replace the masked regions with a domain-agnostic corruption $\phi$: grayscale plus blur, which keeps rough shape and removes texture and color. The generator receives the corrupted image and a mask channel and must restore the original:
$$ \mathcal{L}_{\text{mask}}=\mathbb{E}_{y}\big\|\,G\big((1-m)\odot y+m\odot\phi(y),\,m\big)-y\,\big\|_1 . $$
This term replaces CUT's plain identity loss. The adversarial and PatchNCE losses on source images are kept as in CUT.
Translation at test time. We apply the same corruption to the source's own anchor regions, those with high negative $s$, and set the mask channel accordingly. Translation then reduces to the infilling task the generator practiced on real targets. We also test plain, uncorrupted inputs to see whether the benefit survives without test-time masking.
Where the method should and should not help. On Cityscapes labels→photo nearly every patch differs between domains, so anchor masking should collapse toward uniform masking. We predict no gain there and use it as a built-in sanity check. Gap 5 in the file notes that cycle constraints break when domains carry different amounts of information. Our masks offer a natural way to exclude asymmetric regions from consistency losses, but we treat this as secondary and test it only on Apple→Orange.
Baselines
Do-nothing reference. The naive reference point is CUT as published, with uniform random patch sampling and its ordinary identity loss. It is reported at FID 45.5 on Horse→Zebra and 24.7 mAP on Cityscapes in CUT. We retrain it in our codebase so all comparisons share data splits, resolution, iterations, and seeds.
Strongest image-quality competitor. PUT-3 from Exploring Negatives in Contrastive Learning for Unpaired Image-to-Image Translation reports FID 33.82 on Horse→Zebra and 24.86 mAP on Cityscapes. It is the strongest competitor because it already improves CUT by being selective about patches, which is our own lever applied to negatives instead of masks. We run it from its released settings and give it the same tuning budget we give ourselves: a fixed grid over loss weights and $K$/top-$K$, three seeds each. Beating it means lower mean FID than our reproduced PUT-3 at matched compute, and in any case not worse than 33.82, together with lower background masked-LPIPS. Since our method changes only the identity term, we also test it stacked on PUT's negative ranking.
Strongest background-preservation competitor. BFCUT reports masked-LPIPS of 34.6% vs. 50.5% for CUT on Horse→Zebra in Background-Focused Contrastive Learning. It is the baseline for the bleed question, run under the same budget. Beating it means lower background masked-LPIPS at equal or better foreground FID.
Isolating the key ingredient. The ablations decide the claim. We compare uniformly random masks at the same rate $r$ and at 60% and 75%. We substitute BFCUT's ImageNet heat map for our log-odds score. We substitute Grad-CAM from a binary X-vs-Y classifier for our score. We use masks from an off-the-shelf COCO segmenter where the category exists. Finally, we zero-fill the masked regions instead of applying the shape-preserving corruption $\phi$.
For ManiCups and Yosemite there is no reported result found on Lacuna with exact numbers for any baseline. All numbers there come from our own reproductions.
Why not just…?
Why not just mask image patches uniformly at a high rate? This is the most serious threat. Uniform Masking Prevails shows selective masking advantages disappearing at 60–75% rates in vision-language pretraining. The counter-evidence is either textual (Difference-Masking's 74.04% vs. 63.74%) or about contrastive query sampling (BFCUT), not reconstruction. So the file does not settle the question. Our uniform-mask ablation at matched and high rates does. If uniform masking at 60–75% matches anchor masking on FID and background masked-LPIPS, we will report that the gain comes from masked identity training itself and drop the keyword-scoring claim.
Why not just use an off-the-shelf segmenter to mask the foreground? On Horse→Zebra and Apple→Orange this may work about as well as our scores, and we expect it to. That is why it is an ablation rather than a straw man. The argument against it is scope. A COCO segmenter has no category for "coffee versus milk inside a cup" (ManiCups) or "foliage that will become snow" (Yosemite). There, the region that must change is a sub-object attribute or a scene property. Contrast-GAN's need for such masks is exactly what restricts it. We predict parity on object datasets and an advantage on state and season changes. Parity everywhere would mean the vocabulary step adds nothing beyond a segmenter wherever one exists.
Why not just train a source-vs-target classifier and use its saliency map? This is the closest cheap substitute, because our log-odds score is itself a patch-level domain classifier. The difference is that a classifier's saliency tends to peak on the few most discriminative pixels. The vocabulary statistic plus neighbor spreading is designed to cover the whole divergent region, as Difference-Masking's neighborhood step does for text. We test this directly. Before any translation training, we measure overlap of each mask type with ground-truth regions on Cityscapes labels and on segmenter outputs used for evaluation only. During training, we run the Grad-CAM ablation. If Grad-CAM matches, the simpler scorer should be adopted.
Proposed Experiments and Decisive Checks
We will train on Horse↔Zebra, Apple↔Orange, Cityscapes, Yosemite summer↔winter, and ManiCups at CUT's standard resolution and schedule. We report FID, Cityscapes per-pixel accuracy and mAP, background masked-LPIPS as defined by BFCUT, and FID computed on foreground crops. BFCUT showed that whole-image FID can reward texture bleed, so we never rely on it alone.
The first decisive check is the matched-rate comparison between anchor-biased and uniform masks. It answers whether placement matters in generation, the open question raised by the file's Gap 4. The second is the domain-type split: we should see parity with segmenter masks on object datasets, gains on ManiCups and Yosemite, and no gain on Cityscapes, where everything differs. If gains appear on Cityscapes, the mechanism is not the one we claim. The third is cost. We report the wall-clock time of clustering and scoring against translation training, since keyword extraction added 19–39% to BERT-Base training in Do not Mask Randomly. We also report whether masked identity training reaches CUT's final FID in fewer iterations, mirroring the convergence gains claimed for NB-MLM and MSLM. A clean result would show lower foreground FID and background masked-LPIPS than both PUT-3 and BFCUT, a clear margin over uniform masking at every rate, and overhead small relative to training.
References
- Uniform Masking Prevails in Vision-Language Pretraining
- Difference-Masking: Choosing What to Mask in Continued Pretraining
- Knowledge Enhanced Masked Language Model for Stance Detection
- NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis
- Contrastive Learning for Unpaired Image-to-Image Translation
- Background-focused contrastive learning for unpaired image-to-image translation
- Exploring Negatives in Contrastive Learning for Unpaired Image-to-Image Translation
- AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networks
- Generative Semantic Manipulation with Mask-Contrasting GAN
- Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords
- Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER