Symmetrized Patch Alignment for Unpaired Translation Under Shape Change

Unpaired image translation methods keep the content of an image by assuming that a patch at position (x, y) in the input should still be at (x, y) in the output. Here an image is turned into another visual domain (simulation to real, horse to zebra) without matching training pairs. The assumption holds when only texture changes. It fails when the target domain also needs a different shape, as when a rendered fruit fly must become a real fly with different leg proportions. We propose to replace the fixed-position rule with an alignment step borrowed from statistical machine translation. Each training step aligns input patches to output patches twice, once in each direction, with a penalty on large jumps. The two alignments are then merged with the classical intersect-then-grow procedure. The merged alignment does two jobs. It tells a contrastive loss which output patch should resemble which input patch, and it carries keypoint labels from synthetic images onto translated ones, so a pose estimator can be trained for the real domain.

The practical payoff is sim-to-real pose estimation without foreground masks. The strongest existing method for this task warps silhouettes with an explicit deformation field. It works well, but it needs clean masks and fails in clutter. If an alignment computed directly between image patches can stand in for that deformation field, the same keypoint transport becomes available in cluttered scenes. The alignment is also discrete and can be inspected: one can read off which input region went where.

Core Claim

Contrastive Unpaired Translation (CUT) (Contrastive Learning for Unpaired Image-to-Image Translation) pulls each output patch toward the input patch at the same coordinate and pushes it away from other input patches, using a contrastive (InfoNCE) loss. This gave large gains over cycle-consistency methods: FID 45.5 versus CycleGAN's 77.2 on Horse→Zebra. That fixed-coordinate rule, however, actively penalizes the generator for any spatial deformation. Our claim is that a structured correspondence can replace it, provided the correspondence is computed between the input and the generator's own output, made symmetric, and given a locality prior. This removes the penalty without letting the generator cheat by shuffling patches. The same correspondence also transfers keypoints well enough to approach mask-based deformation methods on their own benchmarks and to beat them once masks become unreliable.

Each ingredient has a specific role, taken from phrase-based translation (Statistical Phrase-Based Translation). A single directional alignment can map many patches to one, so it collapses. Intersecting the two directions keeps only confident one-to-one anchors. Growing from those anchors to neighboring points that appear in the union of the two directions recovers many-to-many structure, which is needed when one limb segment stretches over several patches. Phrase-consistent blocks then supply region-level positives. A block is consistent when its input patches align only inside its output patches, and the reverse. These blocks give a coarser training signal than single patches, in line with Koehn et al.'s finding that short multi-unit phrases beat single-word mappings.

Mechanism

Let $f(\cdot)$ denote the generator encoder's patch features. Let $u_i$ be the features of input patch $i$ at grid position $p_i$, and $v_j$ the features of output patch $j$ at position $q_j$. The input-to-output directional alignment assigns each $i$ to

$$a(i) = \arg\max_j \; \cos(u_i, v_j) - \lambda \,\| q_j - (p_i + \bar{d}_i) \|_2 ,$$

where $\bar{d}_i$ is the average displacement already assigned to $i$'s grid neighbors in the previous pass and $\lambda$ sets the strength of the distortion cost. This is the image analogue of the jump penalty in alignment models: shapes may bend, but neighboring patches should move together. The output-to-input direction is computed the same way. The two are merged by intersection followed by grow-diagonal expansion over the 8-neighborhood. The merged alignment $\mathcal{A}$ replaces the identity map in CUT's loss. For an output patch $j$, the positive is the mean feature of the input patches aligned to it, and the negatives are input patches outside a radius around them. We add a second, region-level loss of the same form over phrase-consistent blocks of up to 3×3 patches. The 3×3 cap follows Koehn et al.'s observation that gains saturate around phrase length 3.

Alignment runs without gradients; only the contrastive loss backpropagates. The alignment is initialized to the identity, so training begins exactly as CUT does. The distortion term and the target-domain discriminator together block the degenerate solution, a generator that shuffles patches so alignment trivially succeeds. Shuffled outputs are both unrealistic and expensive under the jump cost.

For keypoint transport, each synthetic keypoint is mapped through the merged alignment, using the alignment centroid at the patch level and refining with feature correlation inside the matched patch. The pose estimator is then trained on the translated images with the transported labels. Adversarial domain adaptation enters at this stage, both as a baseline and as an optional add-on: a gradient-reversal domain discriminator on the pose network's features, as described in FAN-Net: Fourier-Based Adaptive Normalization For Cross-Domain Stroke Lesion Segmentation Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation. We report results with and without it, to test whether pixel-level alignment makes feature-level alignment unnecessary.

Relation to Prior Work

Three lines of work motivate or constrain the design. CUT supplies the loss we modify; its fixed-coordinate positive is the specific weakness we target. Li et al. (Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals) supply the evidence that shape change matters. They split translation into a spatial transformer, a dense per-pixel warp of silhouettes, and a texture generator, and they transport keypoints through the explicit warp. This raised fly PCK@15px from 72.9% (CycleGAN) to 84.7% and worm PCK@5px from 45.3% to 90.3%. Our method keeps their keypoint-transport idea but drops the silhouette requirement and the continuous flow field in favor of a discrete patch alignment. CyCADA (CyCADA: Cycle-Consistent Adversarial Domain Adaptation) shows that pixel-level translation plus feature-level adversarial alignment and a frozen-classifier consistency loss helps adaptation (SVHN→MNIST 67.1%→90.4%). However, it depends on cycle consistency, which assumes geometry is preserved, and it requires staged training. We do not use its frozen source classifier, since our grounding comes from correspondence rather than predicted labels.

On the alignment side, MBA (Geometry-aware domain adaptation for unsupervised alignment of word embeddings) is a constraint as much as a motivation. It shows that entropic Gromov-Wasserstein transport can fail completely on structurally distant spaces (0.0% versus 25.1% on English-Hindi) and that bidirectional, mass-preserving constraints help. That warns us that a plain optimal-transport matcher may be fragile, but it is evidence from word embeddings, not images, so we test the question directly instead of assuming it. The Segment-Phrase Table (Segment-Phrase Table for Semantic Segmentation, Visual Entailment and Paraphrasing) earlier borrowed phrase tables for vision, linking text phrases to image segments with pre-deep components. It addressed neither translation between image domains nor learned generators. Patch Your Matcher (Patch Your Matcher) steers translation with a frozen geometric matcher and a discriminator that judges correspondences. It is trained on synthetic paired data (MD-Syn) and targets cross-modal matching rather than shape-changing unpaired translation. To our knowledge, no work in this file computes symmetrized, distortion-penalized alignments between a generator's input and output to choose contrastive positives and transport labels.

Research Program

The first study is a controlled check of alignment quality. Synthetic fly renderings are paired with copies warped by known random smooth fields, so the true correspondence exists. On these pairs we measure alignment precision and recall for single-direction argmax, intersection only, union, grow-diagonal, and Sinkhorn transport, each with and without the distortion term. This mirrors how alignment error rate was used to choose symmetrization heuristics in translation. It tells us whether the merge rule matters before any generator is trained.

The second study trains the full translator on the fly, worm, and zebrafish sim-to-real data of Li et al. A pose estimator is trained on translated images with transported keypoints and evaluated on real annotated frames. The third study is a sanity check that the method does no harm when geometry is preserved: Horse→Zebra (FID) and Cityscapes label→photo (segmentation mAP on outputs), where fixed-coordinate CUT should already be near-ideal. The fourth study probes the mask dependence. We paste real cluttered laboratory backgrounds behind the fly renderings, or degrade the silhouettes with standard foreground estimators, and compare how quickly each method degrades.

Baselines

The naive reference is a pose estimator trained on raw synthetic images and tested on real ones, with no translation. Next to it we place CycleGAN-based translation with keypoints copied at fixed coordinates, which Li et al. report at 72.9% PCK@15px on flies and 45.3% PCK@5px on worms. All learned baselines share our pose architecture, image resolution, training images, and the same hyperparameter search budget, set as a fixed number of configurations per method selected on a held-out synthetic validation split.

The strongest existing competitor is Li et al.'s deformation-aware translation, at 84.7% PCK@15px on flies (against an 88.8% manual-supervision oracle) and 90.3% PCK@5px on worms. It is strongest because it is the only method in the file built for shape change with keypoint transport. We will use their reported numbers and, where code allows, rerun it in our pipeline. On the original clean-silhouette benchmarks, we count matching it within two PCK points without masks as success. Beating it outright is expected only in the cluttered and degraded-mask setting, and there we require a margin larger than seed-to-seed variation over three seeds. On zebrafish we found no reported result on Lacuna for this method, so we will run it ourselves.

CUT with fixed-coordinate positives and fixed-coordinate keypoint copying isolates the effect of replacing the identity map. No reported result was found on Lacuna for CUT on the fly or worm data. Its file-reported figures are Horse→Zebra FID 45.5 and Cityscapes 24.7 mAP, and on those our method should not be worse by more than noise. CyCADA, with feature-level adversarial adaptation on the pose network, is the adversarial domain adaptation competitor. No reported result was found on Lacuna for CyCADA on animal pose.

Two ablations isolate the key ingredient. The first swaps grow-diagonal symmetrization for Sinkhorn entropic transport between patches, with the same distortion cost added to its cost matrix. The second removes the distortion term. The phrase-level loss is also ablated separately.

Why not just…?

Why not just use CUT and accept small shape errors? CUT's loss rewards keeping every patch in place, so on worms, where the synthetic and real body shapes differ substantially, it should behave like the fixed-coordinate CycleGAN baseline that Li et al. found reaches only 45.3% PCK@5px. That figure is for CycleGAN, not CUT, so the CUT-on-worm run in our baselines is what settles this. If CUT comes close to the deformation methods, the premise of this proposal is wrong, and we will report that.

Why not just match patches with Sinkhorn optimal transport instead of a 2003 heuristic? This is the real threat. Entropic transport is differentiable, standard in vision, and gives a soft many-to-many matching. MBA's English-Hindi collapse of Gromov-Wasserstein is suggestive, but it comes from word embeddings and from a structure-only objective, not from feature-cost transport between image patches. The controlled warped-pair study and the full-pipeline ablation compare the two directly, with the same distortion cost. If Sinkhorn equals grow-diagonal, the contribution becomes "structured, distortion-penalized correspondence as CUT positives and label transport," and the translation heuristic is just one usable instantiation. We would state that plainly.

Why not just use Li et al.'s deformation field, or plain mutual nearest neighbors? The deformation method needs clean silhouettes, which its authors identify as the obstacle in cluttered scenes. The degraded-mask study measures how much that costs. Mutual nearest neighbors is exactly our intersection step without growing. The controlled study measures its recall, which we expect to be low on stretched limbs, the many-to-many case that growing is meant to recover. If intersection alone performs as well as growing, we will drop the grow step.

Decisive Checks

The proposal succeeds if three things hold. First, grow-diagonal alignment with the distortion cost beats both single-direction alignment and intersection on recall of the known warps in the controlled study. Second, the full method lands within two PCK points of 84.7% on flies and 90.3% on worms without silhouettes. Third, it degrades clearly less than the deformation-aware method as masks are corrupted. It fails if fixed-coordinate CUT is already competitive on the animal data, or if Sinkhorn matching does as well everywhere. In that second case, the evidence would favor framing the work as correspondence-driven contrastive translation without the machinery borrowed from statistical alignment.