Unpaired image translation turns a horse photo into a zebra photo, or a summer scene into a winter one, without any matched before-and-after pairs to learn from. Current systems balance three pulls with hand-picked constants. One pull is realism, judged by a discriminator, a network trained to tell real target images from generated ones. The second is content preservation: the output should keep the input's layout. The third is randomness: how much the translation is allowed to vary from one sample to the next. The best recent methods treat translation as a short chain of noisy steps. In these methods the randomness is a single "entropy" constant, and that constant is known to be treacherous. If it is too small, training is unstable. If it is too large, the output drifts away from the input's structure.
This note proposes to stop choosing these constants. Instead, the user states budgets: "keep at least this much patch-level correspondence with the input," and "keep this much randomness at step $t$ of the chain." The constants become Lagrange multipliers, which are adjusted automatically during training so that each budget is met. This is the same move maximum-entropy reinforcement learning uses when it tunes its temperature to hit a target policy entropy. The claim to test is concrete. One set of normalized budgets should transfer across datasets and match or beat per-dataset grid searches over fixed weights at equal compute. A per-step entropy schedule should also resolve a live disagreement in the literature about whether good unpaired couplings should have high entropy or low entropy.
Core Claim
Three results suggest that the loss weights of a multi-step unpaired translator are all multipliers of one constrained problem.
The first is A Lagrangian Perspective on Latent Variable Generative Models. It shows that the fixed weights in VAE, GAN and CycleGAN-style objectives are dual multipliers held constant. Learning the multipliers to meet a reconstruction budget (their LagVAE) Pareto-dominates fixed weightings. That result was shown only on MNIST.
The second comes from the Schrödinger bridge literature. A Schrödinger bridge is the most random path between two distributions that still connects them, which is entropy-regularized optimal transport. Here the entropy constant $\epsilon$ plays exactly the role that temperature plays in maximum-entropy RL. Both Schrodinger Bridge Flow for Unpaired Data Translation and Unpaired Image-to-Image Translation via Neural Schrödinger Bridge treat $\epsilon$ as a fixed scalar, and the research file records that bridge performance is highly sensitive to it.
The third is the maximum-entropy RL work. Reinforcement Learning with Deep Energy-Based Policies shows that an explicit entropy term keeps policies covering several good modes; DDPG collapses to one goal where soft Q-learning finds all four. Entropy-Preserving Reinforcement Learning shows that entropy should follow a controlled trajectory rather than a fixed penalty.
Putting these together, we write a $T$-step translator with stochastic transitions $\pi_\theta(x_{t+1}\mid x_t)$ as a constrained program:
$$\min_\theta \; \mathcal{L}_{\text{adv}}(\theta) \quad \text{s.t.} \quad \mathcal{C}_t(\theta) \le c, \;\; \mathcal{H}_t(\theta) \ge h_t \quad \text{for } t=1,\dots,T.$$
The symbols are as follows. $\mathcal{L}_{\text{adv}}$ is the adversarial realism loss. $\mathcal{C}_t$ is a normalized patch-contrastive content loss at step $t$, built from the multilayer InfoNCE loss of Contrastive Learning for Unpaired Image-to-Image Translation. $\mathcal{H}_t$ is an estimate of the conditional entropy of the step-$t$ transition. $c$ and $h_t$ are user budgets.
Training alternates generator steps on the Lagrangian with projected dual ascent:
$$\lambda_t \leftarrow \max\big(0,\lambda_t + \eta(\mathcal{C}_t - c)\big), \qquad \beta_t \leftarrow \max\big(0,\beta_t + \eta(h_t - \mathcal{H}_t)\big).$$
Here $\lambda_t$ replaces UNSB's fixed contrastive weight, $\beta_t$ replaces its fixed $\epsilon$ at each step, and $\eta$ is a dual step size.
The new part is not "use Lagrange multipliers," which LagVAE already did. It has two pieces. First, the entropy constant of a bridge becomes a per-step constraint with a schedule. Second, this makes testable a reconciliation between two views. Learning to Match Unpaired Data with Minimum Entropy Coupling (DDMEC) argues that the right final coupling has minimal joint entropy. Bridge methods instead keep paths noisy. A decreasing schedule $h_1 > \dots > h_T$ allows both: the chain explores early and nearly commits by the last step.
Research Program
We build on UNSB because it already has the right structure: a few discrete stochastic steps, each trained with an adversarial loss, a contrastive loss and an entropy-weighted transport term. UNSB reports its best results at three to five steps.
Content constraint. The contrastive loss is normalized so that budgets mean the same thing across datasets. We express $\mathcal{C}_t$ relative to two reference points: the InfoNCE loss of the identity map, which is the best achievable correspondence, and that of a random target image, which is chance. The budget $c$ is then a fraction between these.
Entropy estimate. The entropy term is the most technically fragile piece. Bridge transitions inject Gaussian noise of known variance, so part of $\mathcal{H}_t$ is analytic. The generator's own stochasticity, from its injected noise, is estimated with a nearest-neighbor entropy estimator on a fixed feature space, computed over several samples per input. We will report how well this proxy tracks LPIPS diversity across samples of the same input. If the estimator is too noisy to drive a multiplier, the project reduces to the content constraint alone, and we will say so.
Analysis. We study three entropy schedules: constant, linearly decreasing to a small $h_T$, and decreasing to $h_T \approx 0$. The near-zero case is the minimum-entropy limit. We measure where each lands on the realism–content–diversity surface.
Framing constraint from theory. On The Classification-Distortion-Perception Tradeoff proves, for restoration with a known degradation, that distortion, realism and classifiability trade off along a frontier. It also proves that restoration cannot raise Bayes classification accuracy above that of the degraded input. Translation is not restoration, so the theorem does not apply directly. It still shapes the proposal in two ways. Budgets are the natural way to choose a point on such a frontier. And we will not claim that our translations improve semantic utility. Downstream segmentation scores are reported only as a measure of preserved content.
Baselines
Naive reference points. The do-nothing reference is the identity map. It scores perfect content and the worst realism, and it anchors the normalization. A second naive point is SDEdit, which noises the input and denoises it with a target-domain diffusion model. UNSB reports SDEdit at FID 97.3 on Horse→Zebra, which shows the cost of routing through pure noise.
CUT. Contrastive Learning for Unpaired Image-to-Image Translation is a one-step translator with an adversarial loss plus patchwise InfoNCE. It reports Horse→Zebra FID 45.5 against CycleGAN's 77.2, and Cityscapes mAP 24.7 against 20.4. It represents the strongest single-step fixed-weight design.
Strongest competitor: UNSB with its published weights and $\epsilon$. Unpaired Image-to-Image Translation via Neural Schrödinger Bridge reports Horse→Zebra FID 35.7 and Summer→Winter FID 73.9, against CUT's 45.5 and 84.3. It is the strongest competitor because our method is UNSB with its constants replaced, so any gain is attributable to the change. The comparison uses the same backbone, step count, data, number of iterations and batch size.
UNSB with an equal-budget grid search. This is the fairest version of the skeptic's alternative. We grid-search UNSB's contrastive weight and $\epsilon$ per dataset, with total GPU hours equal to our method's training plus its budget selection. Our budgets are chosen once, on Horse→Zebra only, and then frozen for every other dataset.
DDMEC. Learning to Match Unpaired Data with Minimum Entropy Coupling fine-tunes diffusion models with policy gradients toward minimum-entropy couplings. It reports CelebA-HQ male→female FID 34.86 against SDDM's 44.37. We compare against it on that task using its reported number. Retraining it at equal compute is likely infeasible given its cost, so we will flag the comparison as unequal.
Ablations. Three ablations isolate the key ingredient. In the first, the content multiplier is dual-adjusted and $\epsilon$ is fixed. In the second, $\epsilon$ is dual-adjusted and the content weight is fixed. In the third, the entropy schedule is replaced by a single entropy target shared across steps.
What counts as winning. Beating the strongest baseline means one of two outcomes. On Horse→Zebra, the method lowers FID below 35.7 at matched content preservation and diversity. Or, on the held-out datasets, the frozen budgets match or beat the per-dataset grid-searched UNSB at equal total compute.
Why not just…?
Why not just grid-search the weights and $\epsilon$? This is the real threat. On any single dataset, an exhaustive sweep can find a point as good as what dual ascent finds, because the multiplier at convergence is just some constant. The proposal does not claim better single-dataset optima. It makes three other claims. Budgets stated in normalized units should transfer to new datasets where swept weights do not. The per-step schedule $\beta_t$ is a $T$-dimensional object that a sweep explores poorly. And the user can name the content level they want. The equal-budget sweep baseline settles this directly: if swept UNSB matches frozen-budget translation on Summer→Winter, AFHQ and Cityscapes, the transfer claim is false.
Why not just apply LagVAE's dynamic multipliers to UNSB's existing loss? That is the content-multiplier-only ablation. A Lagrangian Perspective on Latent Variable Generative Models already supplies the method. What it leaves out is the entropy constant, which the research file identifies as the parameter bridge methods are most sensitive to. If the content-only ablation captures nearly all the gain, the entropy component and its RL motivation are unnecessary, and the paper becomes a scaled-up LagVAE result.
Why not just fine-tune with RL and an entropy bonus, as DDMEC plus REPO would suggest? Policy-gradient fine-tuning of diffusion trajectories works for coupling: DDMEC reports 66.3% cell-type accuracy on PBMC against 61.7% for scTopoGAN. But the file notes its high compute cost and its static reward balance. REPO and ADAPO from Entropy-Preserving Reinforcement Learning were validated only on language-model reasoning. Our method takes the one idea from RL that matters here, controlling entropy to a target, without sampling-based gradient estimation. The CelebA-HQ comparison against DDMEC's 34.86 will show whether the cheaper route gives up quality.
Why not just use CUT, since it is one step and cheap? UNSB's 10-point FID gain over CUT on Horse→Zebra suggests that multi-step transport matters. Our method inherits that cost, roughly 3–5 network evaluations, and we will report wall-clock time alongside FID.
Proposed Experiments
The main benchmarks are Horse→Zebra and Summer→Winter, which have reported FIDs for UNSB and CUT, and Cityscapes labels→photo, where CUT reports 24.7 mAP. No reported UNSB result on Cityscapes was found on Lacuna. We add AFHQ cat→dog, where no reported FID was found on Lacuna for any of the listed methods, and CelebA-HQ male→female, where DDMEC reports 34.86.
Metrics are FID for realism, held-out patch InfoNCE and Cityscapes mAP for content, and LPIPS between samples of the same input for diversity. We also report constraint satisfaction: how close the measured $\mathcal{C}_t$ and $\mathcal{H}_t$ end up to their budgets.
There are three decisive checks.
- Transfer. Budgets fixed on Horse→Zebra are reused unchanged elsewhere, against the equal-compute swept UNSB.
- Controllability. Sweeping the budget $c$ should trace a monotone realism–content curve. That curve should dominate the curve traced by sweeping UNSB's fixed weight, which is the scaled-up version of LagVAE's MNIST claim.
- Entropy schedule. We compare constant, decreasing and near-zero final entropy on AFHQ, where one-to-many translation is natural. If the near-zero schedule loses diversity without improving FID or content, the minimum-entropy view is wrong for visual translation. If it improves content at equal FID, a decreasing schedule really does reconcile the minimum-entropy and maximum-entropy positions.
Training stability across five seeds, with the variance of the multiplier trajectories, is reported throughout.
References
- A Lagrangian Perspective on Latent Variable Generative Models
- Schrodinger Bridge Flow for Unpaired Data Translation
- Unpaired Image-to-Image Translation via Neural Schrödinger Bridge
- Reinforcement Learning with Deep Energy-Based Policies
- Entropy-Preserving Reinforcement Learning
- Contrastive Learning for Unpaired Image-to-Image Translation
- Learning to Match Unpaired Data with Minimum Entropy Coupling
- On The Classification-Distortion-Perception Tradeoff