Global Correspondence Search Inside High-Resolution Skip Connections

Encoder-decoder image generators, such as the U-Net, copy fine detail from the input to the output through skip connections. A skip connection is a shortcut that hands an encoder feature map straight to the decoder layer of the same resolution. This works only if pixel $(x, y)$ in the input corresponds to pixel $(x, y)$ in the output. When the output is a re-posed person, a rotated chair, or an undistorted fisheye photo, that assumption fails. The copied detail lands in the wrong place and the decoder produces ghosting and tearing. The standard fix warps encoder features before handing them over. The warp is predicted coarse-to-fine: a rough displacement is estimated at low resolution and then refined by small, local convolutional corrections at high resolution. Local filters can only see a few pixels around each location. So if the coarse stage picks the wrong limb, or loses a chair leg that is thinner than one coarse pixel, nothing at the fine scale can find the right source location. The fine scale is exactly where the skip connection carries the detail worth copying.

This note proposes to make the fine-scale alignment step search the whole image instead of a local window, at linear-times-side-length cost. The search is organized so that it can correct a wrong coarse guess rather than only polish a right one. The mechanism is an alternating one-dimensional search. Each output pixel scores all encoder positions along the row, then along the column, passing through its current best-guess source location. Every refinement step can then jump arbitrarily far along one axis while keeping hard, detail-preserving sampling at the end. The claim is testable with ground-truth correspondences. Coarse-to-fine local alignment should fail sharply once displacement is large and the structure is thin, and the global search should not.

Core Claim

Existing geometry-aware skip connections treat alignment as coarse-to-fine refinement with local estimators. That makes errors committed at coarse resolution unrecoverable, and those errors concentrate on thin or small structures and ambiguous correspondences, such as symmetric limbs or repeated texture. A fine-scale aligner with a global but structured search space can recover them. It should measurably reduce skip-feature misalignment and the downstream artifacts on thin structures and large displacements, at cost comparable to existing multi-scale deformation modules.

The motivating evidence comes from three papers. PCN for fisheye rectification predicts appearance flow with local convolutional blocks to pre-correct skip features, and names the failure it is fighting "spatial pollution." ID-Unet uses soft attention for correspondence only at coarse resolution and residual local flow at fine resolution, and it reports thin-structure ghosting. Hou et al.'s depth-guided skips also report breaking thin structures such as chair legs. Show Your Face constrains the design. It found that badly aligned skips are worse than no skips, so the new module must know when it has failed and fall back to decoder synthesis rather than paste wrong detail.

Mechanism

The module sits in each skip connection at every resolution, including the full-resolution ones. It keeps a current source estimate $s_p = (u_p, v_p)$ for each output pixel $p$, initialized from the next-coarser level's upsampled flow. One refinement step has two passes.

In the row pass, $p$ forms a query $q_p$ from decoder features. It then scores the encoder keys $k$ at every position along the horizontal line $v = v_p$ and updates $u_p$:

$$u_p \leftarrow \sum_{u'} \mathrm{softmax}_{u'}\!\left(\frac{q_p^\top k_{(u',v_p)}}{\tau} + b(u'-u_p)\right)\, u',$$

Here $\tau$ is a temperature and $b$ is a learned bias on distance from the current guess, which lets the model prefer, but not require, small corrections. The column pass does the same along $u = u_p$ to update $v_p$. Two to three alternations let the estimate travel anywhere in the image. The cost per pass is $O(N(H+W))$ for an $H \times W$ map with $N = HW$ pixels, since each pixel scores one row plus one column.

After the search, encoder features are hard-sampled bilinearly at $s_p$, so high-frequency detail is copied and not averaged. The peak attention weight serves as a confidence $c_p$ that gates between warped encoder features and the decoder's own features. That gate is the explicit fallback for occluded or unfindable regions.

Two ingredients come from existing work. The row-and-column factorization comes from CCNet, which applies it to aggregate segmentation context around the pixel itself. The difference here is that the cross is centered on a moving correspondence estimate and ends in a hard sample, so the operation is a search rather than context pooling. Queries and keys receive appended coordinate channels, following BCN, so the scores can use absolute position. Fisheye periphery is the main case where that matters.

The module differs from each nearby design in a specific way. Deformable GANs restrict skip warps to per-part affine maps from a pose detector. Pose Flow estimates flow with a local FlowNetS and then masks the errors with gated attention rather than re-finding the source. DAS learns deformable offsets for classification backbones but never uses them for alignment inside a skip connection. ViT-V-Net puts global attention only in the bottleneck and leaves the skips as identity copies.

Research Program

The first step builds one shared U-Net generator whose skip connections can be swapped among these options: identity, none, local coarse-to-fine flow in the style of ID-Unet and PCN, dense global correlation at a single mid resolution, and the proposed alternating search. This isolates the alignment module from differences in backbone and training.

The second step builds a correspondence-labeled evaluation. Synthetic fisheye pairs generated from Places2 with a known polynomial distortion model, as in PCN, and rendered ShapeNet or 3D Chair view pairs both give exact ground-truth source coordinates for every target pixel. Skip-level alignment error can therefore be measured directly as end-point error in pixels, not only through final image quality. The analysis bins pixels by two quantities: true displacement relative to the fine-scale estimator's effective receptive field, and local structure width relative to the coarsest stride. The prediction is a specific failure region, large displacement combined with thin structure. Local coarse-to-fine alignment should break there, and the global search should hold.

The third step moves to real benchmarks without ground-truth flow: DeepFashion pose transfer (dataset), MultiPIE face view synthesis, and 3D Chair. The question is whether reduced alignment error turns into better images, especially on limbs and chair legs.

Baselines

The naive reference points are the identity-skip U-Net and the skip-free encoder-decoder, trained in the shared backbone. The file already shows these orderings. On MultiPIE, ID-Unet reports identity accuracy of 0.396 for a standard U-Net and 0.651 for a cVAE. On ShapeNet Chairs, Hou et al. report SSIM of 0.8851 for pixel generation without aligned skips. Reproducing these orderings is a sanity check on the reimplementation.

The strongest existing method is local coarse-to-fine skip alignment. On face and chair view synthesis this is ID-Unet, which reports identity accuracy 0.911 and FID 25.12 on MultiPIE and SSIM 0.847 on 3D Chair. On fisheye rectification it is PCN, which reports PSNR 25.06 and SSIM 0.8732 on Places2. On DeepFashion, Pose Flow reports LPIPS 0.124. Hou et al.'s 0.9256 SSIM on ShapeNet Chairs uses a different protocol from ID-Unet's 3D Chair number and will not be compared to it directly.

Each baseline is run twice. The first run uses the authors' numbers under their protocols. The second reimplements their alignment module in the shared backbone with the same data, the same number of training iterations, and the same hyperparameter-search budget. The alignment modules are matched to within about 20% in FLOPs by adjusting channel width.

The proposal beats the baseline only if three things hold. It must lower end-point error in the large-displacement, thin-structure bins by a margin clearly outside run-to-run variance across three seeds. It must not raise error elsewhere. It must match or improve the reported headline metrics under the original protocols, for example SSIM above 0.8732 on the fisheye benchmark and LPIPS below 0.124 on DeepFashion.

Three ablations isolate the key ingredient. The first centers the cross on the target pixel itself, as in CCNet, instead of on the correspondence estimate. The second replaces the final hard sample with soft attention-weighted averaging. The third removes the confidence gate.

Why not just…?

Why not just trust coarse-to-fine local flow, since the coarse level already has a large receptive field? This is the most serious threat, because for large, smooth motions of big objects it is probably enough. The argument against it is specific. A structure narrower than the coarsest stride has no reliable representation at the level that is supposed to find its displacement, and the local refinement above cannot search far enough to fix the mistake. The ghosting on thin structures that ID-Unet and Hou et al. report is consistent with this, but not proof of it. The binned end-point-error analysis settles the question. If the local baseline matches the proposal in the thin, large-displacement bins, the core claim is wrong.

Why not just do dense global correlation once at a mid resolution and upsample the flow? StyleHEAT found that a 64×64 feature map was the best level for dense flow warping, and dense correlation at that size is affordable. This baseline removes the receptive-field problem but keeps the resolution problem. Anything thinner than a 64×64 cell is still localized only by upsampling. It is included as a baseline at the largest resolution that fits in memory. If it matches the proposal, the conclusion is that one global step at mid resolution suffices and full-resolution search is unnecessary.

Why not just add coordinate channels or oversized kernels to the flow estimator? BCN and ParCNetV2 show cheaply that global receptive fields and absolute-position awareness improve relational and localization tasks. These give the estimator global context, but they still regress a displacement rather than compare features against candidate source locations. It is plausible that regression fails on ambiguous matches, such as the left arm versus the right arm, where explicit comparison succeeds. A skip-connection flow estimator built from ParCNetV2-style oversized depthwise kernels plus BCN coordinate channels, with matched FLOPs, is run as a baseline. The DeepFashion limb crops and repeated-texture fisheye scenes are where the two approaches should separate.

Why not just drop skip connections, or keep identity skips and a bigger decoder? The file's numbers already rule both out on these tasks. On MultiPIE, ID-Unet reports 0.651 identity accuracy for a skip-free cVAE and 0.396 for a U-Net with identity skips, against 0.911 for aligned skips. On fisheye rectification, PCN reports PSNR 22.16 for the unaligned baseline against 25.06 with aligned skips. Both remain as reference points rather than contenders.

Decisive Experiments

The central test is the displacement-by-thinness end-point-error map on synthetic fisheye and rendered chair pairs. The claim survives only if the proposal's advantage over local coarse-to-fine flow and over mid-resolution dense correlation is concentrated in the predicted region and grows with displacement.

The second test is downstream. On 3D Chair and DeepFashion, masked SSIM and LPIPS restricted to thin-structure regions (chair legs from rendered part masks, limbs from pose keypoints) should improve more than whole-image metrics. That would show the alignment gain is not washed out by the decoder.

The third test is the fallback behavior motivated by Show Your Face. On occluded regions with ground-truth visibility from rendering, the confidence $c_p$ should fall. With the gate enabled, the proposal should never be worse than the skip-free model in those regions.

The fourth test measures cost. Latency and memory at 256×256 are reported against ID-Unet's multi-scale deformation modules, since computational overhead is a known weakness of that line of work.