Tagged Back-Translation for Unpaired Image Translation

Unpaired image translation learns to turn horses into zebras, or semantic label maps into street photos, from two separate piles of images with no matched pairs. Without pairs there is no direct target for any input, so methods add indirect constraints. The best known is cycle consistency: translate there and back, and require that the original comes back. Cycle consistency has a known failure. The forward generator learns to hide the source image in faint, high-frequency noise inside its output, so the backward generator can decode it (CycleGAN, a Master of Steganography). Machine translation faced a similar shortage of paired data and solved it a different way. It uses back-translation: a reverse model turns real target sentences into synthetic source sentences, and the forward model trains on the resulting pseudo-pairs. To keep the forward model from imitating the oddities of machine-made inputs, each training input is marked with a provenance token such as [SYNTHETIC] or [AUTHENTIC]. At test time the model is always run in [AUTHENTIC] mode (Facebook AI's WAT19 Myanmar-English Translation Task Submission, Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task).

We propose to carry this recipe into images. Train two one-directional translators. Each one also learns from pseudo-pairs built by the other: a machine-made input matched with a real target image. The pseudo-pairs are built with gradients blocked through the producing model, so neither generator is ever rewarded for hiding information. Every input carries a learned source-side token saying whether it is a real photograph or a machine-made one. The pseudo-pairs give the forward generator something cycle consistency never does: a real, pixel-level target. The tag keeps the artifacts of machine-made inputs from leaking into how the model treats real ones. This makes a cycle loss unnecessary, and with it the incentive to hide information.

Core Claim

The claim has two parts. First, pseudo-paired supervision with stopped gradients can replace cycle consistency in unpaired image translation. It gives stronger content preservation than a contrastive patch loss alone, and it creates no steganographic channel. Second, the provenance tag is the ingredient that makes this work. Untagged pseudo-pair training should be measurably worse, because the generator adapts to the color shifts, blur and residual high-frequency patterns of machine-made inputs, and those statistics are absent from real test images.

The tag is a source-side indicator in the exact sense of the language literature. It describes where the input came from, not what the output should be. That separates it from every image method in the file, all of which condition on the target or on style. StarGAN feeds only a target-domain label (StarGAN). EDIT generates generator weights from a target label and an exemplar image (EDIT). DosGAN uses a pretrained classifier's features as a style code (Unpaired Image-to-Image Translation with Domain Supervision). The closest cyclic work, DoraCycle, does insert special tokens into a cycle-trained multimodal transformer (DoraCycle). Those tokens name characters, though, not data provenance. DoraCycle also needs a paired seed set plus gradient surgery to keep its cycles stable. We use no paired data and no text.

Research Program

Each direction has a generator built on the one-sided contrastive formulation of CUT. CUT trains a single forward generator by pulling each output patch toward the input patch at the same location and away from other patches of the same image (Contrastive Learning for Unpaired Image-to-Image Translation). We implement it in two backbones. One is CUT's convolutional network, where the tag enters through a learned per-channel scale and shift. The other is ITTR's hybrid convolution-and-attention generator (ITTR), where the tag is prepended as an extra token that every patch token can attend to. That is the "prefix switch" used in text and in multi-domain time-series diffusion (Time Series Generation Under Data Scarcity). The forward generator's objective is

$$\mathcal{L}_{AB} = \mathcal{L}_{\text{adv}}\big(G_{AB}(a,\texttt{REAL})\big) + \lambda_{\text{NCE}}\,\mathcal{L}_{\text{NCE}}(a) + \lambda_{\text{BT}}\,\mathbb{E}_{b}\,\big\| G_{AB}\big(\mathrm{sg}[G_{BA}(b,\texttt{REAL})],\,\texttt{BT}\big) - b \big\|,$$

where $a$ and $b$ are real images from the two domains and $\mathcal{L}_{\text{adv}}$ is the usual adversarial loss. $\mathcal{L}_{\text{NCE}}$ is CUT's patch contrastive loss. $\mathrm{sg}[\cdot]$ blocks gradients, so $G_{BA}$ receives nothing from this term. The norm is an L1 plus perceptual distance. REAL and BT are the two provenance tags, the latter short for "back-translated". The reverse generator is trained symmetrically. Pseudo-pairs are regenerated online, so their quality improves as the reverse model improves. This mirrors iterative back-translation.

The gradient block is the structural reason no steganography should appear. In a cycle, $G_{AB}$ is rewarded whenever $G_{BA}$ can read its output, so hiding pays. Here the only pixel-level loss compares an output to a real image, and the model that produced the input is not optimized by that loss. Information that $G_{BA}$ leaks by accident could still be exploited by $G_{AB}$ in BT mode. The tag is meant to confine that exploitation to the mode that is never used at test time.

The study has three parts. The main one compares tagged back-translation against CUT, ITTR, CycleGAN and a filtered CycleGAN on standard benchmarks. The second is a set of mechanism analyses showing what the tag does. These include running real inputs in BT mode and machine-made inputs in REAL mode, measuring how far the two modes' outputs diverge, and testing how separable real and machine-made inputs are to a small probe. The third is a steganography audit that applies the fragility test of Chu et al. to every method.

Baselines

The do-nothing reference is plain CUT, trained with the same backbone, data, iterations and augmentation as our method but with $\lambda_{\text{BT}}=0$ and no tag. It shows what pseudo-pairs add on top of contrastive content preservation. CUT reports FID 45.5 on Horse→Zebra and 24.7 mean average precision (mAP) on Cityscapes label→photo, against CycleGAN's 77.2 and 20.4 (Contrastive Learning for Unpaired Image-to-Image Translation). FID, the Fréchet Inception Distance, measures how far generated images are from real ones in a pretrained network's feature space; lower is better. The mAP score here comes from segmenting the generated photos with a pretrained segmenter.

The strongest existing method in the file for Horse→Zebra is ITTR, which reports FID 33.6 (ITTR). We will retrain ITTR from its published configuration and use the reproduced number as the bar. For our ITTR-backbone variant, beating ITTR means a lower FID than reproduced ITTR across three seeds, at equal training iterations. To keep total compute fair, we also report ITTR trained for the extra wall-clock time our second generator costs. The file reports no Cityscapes number for ITTR, so on Cityscapes the bar is CUT's 24.7 mAP and its reproduction.

For the nuclei microscopy task on Data Science Bowl 2018, the strongest competitor is the DCT-filtered CycleGAN of Focus on Content not Noise. It removes the top half of image frequencies inside the cycle and reports a downstream segmentation F1 of 74.0%, against 68.6% for vanilla CycleGAN. A segmenter trained on our synthetic images must exceed 74.0% F1 under the same segmentation protocol to count as beating it. Vanilla CycleGAN, trained on every benchmark with matched compute, anchors the steganography audit. On aerial↔map translation, we found no reported result on Lacuna for any method.

Three ablations isolate the key ingredient. Untagged back-translation is identical to our method except that no tag is used, which tests the second half of the claim directly. Shuffled tags, assigned at random, test whether gains come from the information in the tag or merely from extra parameters. A version where the tag is a continuous embedding added to the input features, rather than a prepended token, asks whether the finding of How to Encode Domain Information in Relation Classification carries over to images. In that study discrete markers raised Macro-F1 from 36.47 to 38.66 and beat learned dataset embeddings. All variants share one hyperparameter search budget over $\lambda_{\text{BT}}$ and $\lambda_{\text{NCE}}$.

Why not just…?

Why not just use back-translation without a tag? This is the most serious alternative, and the untagged ablation exists to settle it. The case for tags comes from machine translation: models that saw synthetic inputs without a provenance marker absorbed the statistics of machine-made text (Facebook AI's WAT19 Myanmar-English Translation Task Submission). Images give a specific reason to expect the same, since translator outputs carry systematic high-frequency residue (CycleGAN, a Master of Steganography). We have no image-domain evidence yet. If tagged and untagged differ by less than seed variance on all four benchmarks, the tag is unnecessary. The contribution then shrinks to "stop-gradient back-translation replaces cycles", and we will report it that way.

Why not just clean the pseudo-inputs instead? One could low-pass filter or add noise to machine-made inputs so their artifacts disappear, as the DCT filter of Focus on Content not Noise does inside a cycle. Filtering removes genuine fine structure along with artifacts, which hurts on aerial building edges and cell membranes. Noise only hides artifacts rather than telling the model they exist. This too is an empirical question, so we include untagged back-translation with DCT-filtered and with noise-augmented pseudo-inputs as extra baselines. Agreeing to compare against them is part of the proposal.

Why not just rely on CUT or ITTR, whose contrastive loss already preserves content? A patch-correspondence loss preserves where things are but gives no target for what the output pixels should be. On low-to-high entropy tasks such as label→photo, that is exactly where fidelity is lost. Plain CUT is the do-nothing baseline. If pseudo-pairs fail to improve Cityscapes mAP over CUT's 24.7, the main claim fails.

Why not just let the network infer provenance from the pixels? Machine-made inputs may be easy to detect, which would make the tag redundant. We measure this with a linear probe on the generator's first-layer features. Then we compare untagged against tagged training at matched probe accuracy. The machine-translation result suggests an explicit marker helps even when provenance is detectable, but we treat that as a hypothesis, not a given.

Proposed Experiments

The benchmarks are Horse→Zebra (FID), Cityscapes label↔photo (mAP from the pretrained segmenter, plus FID), aerial↔map from the steganography study, and Data Science Bowl 2018 mask→microscopy, scored by downstream segmentation F1 and precision at IoU 0.75. All are run with three seeds.

The first decisive check is whether tagged back-translation beats reproduced CUT on Cityscapes mAP and reproduced ITTR on Horse→Zebra FID. The second is whether tagged training beats untagged, shuffled-tag and cleaned-input back-translation, which tests whether the tag carries the gain.

The third check is the steganography audit. For each method, train a fresh decoder to reconstruct the input $a$ from the output $G_{AB}(a)$. Then measure how much reconstruction quality falls when the output is perturbed with Gaussian noise of amplitude 3/256 or saved as a JPEG. The amplitude 3/256 is the level at which Chu et al. found cycle reconstructions collapse. A large fall means content is held in a fragile hidden channel. Our prediction is a large fall for CycleGAN on aerial→map, a much smaller one for ours, and little difference between our two tag modes on real inputs.

The fourth is the mode-swap test. Real inputs translated in BT mode should look worse, or more artifact-matched, than in REAL mode. If the two modes produce identical outputs, the tag is doing nothing, regardless of what the scores say.