Unpaired image translation systems learn to turn one kind of image into another, such as horses into zebras or maps into aerial photos, without matched examples. They are trained on a weighted sum of losses that pull in different directions. An adversarial loss pushes the output to look like the target domain. A content loss, either a round-trip "cycle" reconstruction or a patch-matching contrastive term, pushes the output to keep the input's structure. Practitioners set the weights by hand, and results are known to be sensitive to them.
This proposal starts from a simple observation. These losses are not global quantities: each produces a gradient at every pixel of the generated image. The fight between "look like a zebra" and "keep the horse's shape" is fierce on the animal's body and nearly absent on the sky. A single scalar weight per loss therefore has to pick one compromise for the whole canvas.
We propose to balance the losses separately at each location of the generator's output, before backpropagating into the generator. We will test whether this removes much of the known sensitivity to loss weights. A second component is a two-level quality estimator, borrowed from the quality-estimation literature, that predicts both an image-level score and a per-pixel error map. We use it to find where each method fails. We also test whether a learned error map is a better balancing signal than the raw gradient conflict.
Core Claim
A large share of the loss-weight sensitivity of CycleGAN-style and contrastive (CUT) translators comes from gradient conflict that varies across space. Resolving that conflict per pixel, in the output image rather than in the network's parameters, should flatten the performance-versus-weight curve. It should also keep or improve peak quality, at close to the cost of ordinary training.
The mechanism is concrete. Let $y = G(x)$ be the generator's output. For each loss $L_i$, compute $g_i(p) = \partial L_i / \partial y(p)$, the gradient of that loss with respect to the output at pixel or patch location $p$. These are dense maps of the same size as the image, and each is obtained from one backward pass through the loss's own network (the discriminator, or the encoder features used by the contrastive loss). No backward pass through the generator is needed to get them.
At each location we combine the maps in two steps. First, we rescale each $g_i(p)$ toward a shared local magnitude, in the spirit of GradNorm. Second, wherever two of them have a negative inner product, we project each onto the normal plane of the other, as in PCGrad. This yields a combined map $\hat g(p)$. The generator is then updated with a single vector-Jacobian product,
$$\nabla_\theta = \left(\frac{\partial y}{\partial \theta}\right)^{\top} \hat g,$$
where $\theta$ are the generator's parameters. The whole method therefore costs one generator backward pass plus cheap per-pixel arithmetic, not the $M$ generator backward passes that parameter-level balancing needs for $M$ losses.
Relation to Prior Work
The balancing methods supply the ingredients and also show the limitation. GradNorm adapts one weight per loss so that gradient norms and training rates stay even. Gradient Surgery for Multi-Task Learning (PCGrad) projects away conflicting components of whole-network gradient vectors. MTAdam keeps separate Adam moments per loss and equalizes magnitudes layer by layer against a chosen anchor loss, at about 20% extra per-iteration compute. All three treat each loss as producing one gradient per parameter tensor, and none was evaluated on cycle-consistent or contrastive unpaired translation. We reuse their rules but apply them per output location, which also sidesteps the generator/discriminator asymmetry: every generator loss meets at the same tensor $y$.
Two quality-estimation lines show that dense local error prediction anchors a global score. In machine translation, CometKiwi and QE BERT pair word-level error tags with sentence scores. In imaging, QCResUNet jointly predicts segmentation-quality scores and voxel error maps, and TempQT predicts a pixel error map to support a blind quality score. QCResUNet also supplies a data recipe we reuse: it harvests realistic failures from under-trained checkpoints.
The closest links to translation are three papers. QGAN adds a global FSIM or generator-feature quality term to CycleGAN with fixed weights. AttentionGAN learns foreground masks that restrict edits but estimate nothing about errors. QE-assisted post-editing couples word-level quality estimation with Nash-MTL balancing for text, and is the nearest conceptual precedent. We differ from all three: the balancing is spatial, it is applied to adversarial and cycle/contrastive image losses, and quality estimation is tested both as a diagnostic and as a candidate balancing signal rather than assumed to help.
One paper constrains the design. CycleGAN, a Master of Steganography shows that cycle generators hide source information as faint high-frequency noise. Our per-pixel cycle-gradient maps should reveal this, but we do not claim to fix it, because a cheaper remedy exists (discussed below).
Research Program
The first stage measures the phenomenon before any fix. We train CycleGAN and CUT with their published weights on Horse→Zebra, Cityscapes labels↔photos, and Maps. At regular checkpoints we log per-pixel cosine maps between adversarial and content gradients at the output. We also compute, on a subset of steps, the same conflicts in parameter space, to check that output-space conflict predicts parameter-space conflict; the Jacobian can in principle mix them. The prediction to test is that conflict is concentrated on the translated object and on high-texture regions, and that the fraction of conflicting pixels drifts over training.
The second stage builds the translation quality estimator. We use a shared encoder with two heads. A global head predicts a per-image score: the segmentation score of the output for Cityscapes, and L1 error to the held-out aligned photo for Maps. A dense head predicts the per-pixel error map against ground truth. Both targets exist only because these benchmarks happen to have pairs, which we use for evaluation only, never for training the translator. Training images come from early, late and differently weighted checkpoints of all methods, following QCResUNet's failure-harvesting approach.
The estimator serves two purposes. It localizes where each method loses quality, for example background over-translation versus object distortion. It also acts as an alternative per-pixel weight in the ablations. We will report its error-map agreement on held-out methods. Following the high-quality ceiling noted in WMT 2020 QE findings, we will also report whether that agreement collapses on near-correct outputs.
The third stage is the method itself, applied to CycleGAN (to both generators' outputs, including the intermediate image in the round trip) and to CUT. It is then evaluated through systematic weight sweeps.
Baselines
The naive reference is each translator with its published fixed weights and ordinary Adam. The file reports CUT at FID 45.5 versus CycleGAN at 77.2 on Horse→Zebra, and 24.7 versus 20.4 mAP on Cityscapes (CUT). It reports AttentionGAN at 68.55 FID on Horse→Zebra (AttentionGAN). We will reproduce these in our codebase and quote the paper numbers beside our reproductions.
The strongest competitor on the weight-sensitivity question is a tuned grid search over the content-loss and identity weights. We give it nine full training runs per dataset, and it is judged both by its best run and by its median run. The proposal gets a single run at default weights, plus the same nine-point sweep, so that its flatness can be measured. Beating grid search means two things. First, our single default run is within noise of grid search's best on FID and mAP. Second, across the sweep, the range of weights within 10% of the best FID is at least several times wider.
The strongest existing automatic balancers are MTAdam, PCGrad and GradNorm, run at the parameter level on the generator. They use the same data, iterations and wall-clock budget, and each gets one small tuning sweep of its own hyperparameters. No reported result was found on Lacuna for any of them on unpaired translation. The comparison therefore rests on our runs, and beating them means lower FID or higher mAP at equal compute, plus a flatter sweep.
Two ablations isolate the key ingredient. The first is a spatially uniform version of our rule: the same rescaling and projection, computed from the whole output map pooled into one number per loss. If it matches the per-pixel version, the gain comes from working in output space, not from spatial adaptivity. The second is a mask-swap ablation. It replaces the raw conflict map as the per-pixel weight with, in turn, the learned quality-estimator error map and AttentionGAN's attention mask. This settles whether quality estimation adds anything beyond the gradients themselves.
Why not just…?
Why not just grid-search the weights? It can find a good setting on one dataset. GradNorm reports that a single adaptive run beat a 100-trial grid on its multi-task setting, and MTAdam shows plain Adam dropping from 97.9% to 88.8% on MNIST under poor static weights. Neither result concerns translation, however, so this remains a real threat. The median-versus-best comparison and the sweep-width metric settle it directly. If our method only matches grid search's best without widening the stable range, the sensitivity claim fails.
Why not just apply MTAdam or PCGrad to the generator's parameters? They address magnitude and direction conflict, but only one compromise per tensor. MTAdam also needs an anchor loss and about 20% extra compute. If spatial heterogeneity is small, they should do as well as our method; the conflict-map measurements in the first stage and the head-to-head at matched compute decide this.
Why not just use attention masks or a perceptual loss to protect backgrounds? AttentionGAN's masks fail on occlusions, and QGAN's quality term is global, so neither says where losses conflict. Both nonetheless enter the mask-swap ablation as spatial weights. If attention masks match conflict maps, the simpler signal wins, and we will say so.
Why not just add noise to the intermediate image to stop steganographic cheating? Chu et al. show that 3/256 Gaussian noise destroys hidden-code reconstruction, which makes noise augmentation a cheap fix. That is why we do not pitch our method as an anti-steganography tool. On Maps we will report both, alone and combined, only as a diagnostic.
Proposed Experiments
The decisive check is the weight sweep on CUT and CycleGAN across Horse→Zebra, Cityscapes and Maps, with content weight spanning two orders of magnitude. The measured quantity is the spread of FID and Cityscapes mAP and per-pixel accuracy (CycleGAN reports 0.52 against pix2pix's 0.71 in CycleGAN) across the sweep.
The claim holds if per-pixel balancing narrows that spread markedly relative to both fixed weights and parameter-level balancers, while the spatially uniform ablation does not. The quality estimator then tells us where the recovered quality lives: whether gains come from less background over-translation or from better object structure, checked against ground-truth error maps on Maps and Cityscapes.
A negative result would also be informative. If output-space conflict does not predict parameter-space conflict, or if uniform balancing matches per-pixel balancing, then weight sensitivity in unpaired translation is a global-scale problem, and the existing balancers are the right tool.
References
- GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
- Gradient Surgery for Multi-Task Learning
- MTAdam: Automatic Balancing of Multiple Training Loss Terms
- CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task
- QE BERT: Bilingual BERT Using Multi-task Learning for Neural Quality Estimation
- QCResUNet: Joint subject-level and voxel-level segmentation quality prediction
- Blind Image Quality Assessment via Transformer Predicted Error Map and Perceptual Quality Token
- Quality-Aware Unpaired Image-to-Image Translation
- AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networks
- Quality Estimation-Assisted Automatic Post-Editing
- CycleGAN, a Master of Steganography
- Findings of the WMT 2020 Shared Task on Quality Estimation
- Contrastive Learning for Unpaired Image-to-Image Translation
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks