Shift-Stable Identity in Personalized Face Inpainting

Personalized face inpainting fills a large missing region of a photo, such as eyes hidden by a headset or a face half cut away, so that the result looks like that specific person. It does this by conditioning on reference photos of the same individual. Current systems are scored on how closely the filled face matches the person's identity, and they score well. Nobody has checked whether the answer stays the same when the photo is moved by a pixel or two. Classifiers are known to change their top prediction for up to 30% of images under a one-pixel shift (Azulay & Weiss). That failure points to a version of the same problem here: the network may decide which face to draw partly from where the hole sits on the canvas, rather than from the reference photos.

We propose to measure this "identity drift under translation" and trace it to its architectural sources. We then test a specific fix that resolves a known conflict. The standard cure for shift sensitivity is to low-pass filter before downsampling, but that removes exactly the fine textures (iris pattern, brow hairs, skin detail) that identity depends on. Our design separates the two roles. The coarse path, where the network commits to face layout and identity, is made shift-stable. Fine identity texture is supplied by a full-resolution branch that never downsamples, so it never needs blurring.

Core Claim

Our hypothesis is that identity drift in global-attention inpainters comes from three locatable sources:

  • Aliasing in strided downsampling of the coarse path.
  • The fixed window grid of windowed attention, which decides which tokens can see each other.
  • Border padding, which hands the network an absolute coordinate.

The claim has two parts.

  1. Removing these three sources in the coarse path should cut drift substantially while leaving identity-to-reference similarity unchanged. We plan to remove them with three changes: anti-aliased resampling plus an equivariance penalty, grid-free mask-aware axial attention in place of windows, and padding-free borders.
  2. The identity cost that low-pass filtering normally imposes should be recovered by a stride-free, exemplar-conditioned detail branch.

If drift turns out to be negligible, or if plain anti-aliasing costs no identity, the paper reports that instead. Both are informative results for a field that currently measures neither quantity.

Research Program

Diagnostic protocol. We first build the measurement, since none exists. Shifts are applied jointly to the image and the mask. Integer shifts are produced by re-cropping 512-pixel windows at different offsets from 1024-pixel CelebA-HQ images, so new real content enters at the border instead of wrapping around. Fractional shifts (0.25 and 0.5 pixel) are produced by Fourier phase shifts. For diffusion models, the initial noise is shifted with the image and sampling is deterministic, so stochasticity is not mistaken for drift.

After inpainting, outputs are un-shifted and compared using three metrics:

  • Shift-ID consistency: ArcFace cosine between the output at offset zero and the output at offset δ.
  • Identity flip rate: the fraction of cases where the output's nearest identity in a gallery changes across shifts. This is the generative analogue of the consistency metric of Zhang.
  • Shift PSNR inside the hole: the measure AF-LDM used for latent stability.

Localizing the drift. Three controlled comparisons attribute drift to its sources:

  • Window grid: compare shifts that are exact multiples of the attention window size (the window grid is unchanged) against shifts that are not. A large gap implicates the grid.
  • Border: compare the face near the canvas border against the face centered.
  • Coarse versus fine: swap only the downsampling layers for anti-aliased ones.

Source of the methods. Each source maps to prior work:

  • The window grid and the 512-multiple padding constraint come from MAT. MAT handles large holes by attending only over valid tokens within windows.
  • PATMAT adds person-specific style anchors trained with an ArcFace loss on top of MAT, and inherits that grid.
  • Border-anchor effects motivate the padding analysis. Kayhan & van Gemert showed that zero padding lets filters read absolute position. Alsallakh et al. showed that zero padding creates detection blind spots.

The model. We modify MAT/PATMAT in three ways, each lifted from a different line of prior work.

  1. Grid-free attention. Windowed attention is replaced by a mask-aware version of the recurrent criss-cross attention of CCNet. Each position attends along its row and column, and two passes give every position a global view at $O(N\sqrt{N})$ cost. We restrict keys to valid tokens as MAT does. Criss-cross attention has no window grid, so it is equivariant to integer shifts by construction, given equivariant inputs. It also removes the 512-multiple padding requirement.
  2. Anti-aliased coarse path. The coarse encoder uses blur-before-subsample downsampling (Zhang) and low-pass-wrapped nonlinearities (Alias-Free ViT). It is trained with an explicit equivariance loss of the kind AF-LDM found necessary, because generative objectives otherwise route around the filters.
  3. Stride-free detail branch. A full-resolution branch with no strides takes the reference exemplar and injects high-frequency detail into the last decoder stages through cross-attention from target pixels to exemplar features. Two properties make this branch shift-stable without blurring. Its operations are stride-free, so there is no subsampling to alias. The exemplar keys carry no target-canvas coordinate, so each output pixel depends only on its own query features and the exemplar.

Relation to prior work. This differs from Alias-Free ViT, which reaches shift invariance by collapsing spatial attention into channel covariance and therefore cannot route localized exemplar detail. It also differs from AF-LDM, which anchors attention to a video reference frame and has no identity conditioning. PVA injects FaceNet identity tokens through parallel cross-attention into a frozen Stable Diffusion UNet. It is used here as a diagnostic subject, alongside MAT and PATMAT, to test whether drift is specific to the GAN-transformer design or general.

Baselines

Unmodified PATMAT-C. The naive reference point is PATMAT-C trained and evaluated under our protocol on CelebA-HQ and its curated seven-person set. PATMAT reports ArcFace cosine 0.66 and FID 23.7, against base MAT at 0.35 and 22.9. That is the identity bar we must hold. There is no reported result found on Lacuna for shift consistency of PATMAT, MAT, or PVA, so their drift numbers will be produced by our protocol.

PVA. The strongest identity-retention method in the file reports ID similarity 0.741 and FID 8.22 on CelebAHQ-IDI. It will be run with deterministic sampling and shifted noise, using its published 40-step per-identity tuning, to see whether a diffusion pipeline drifts more or less than MAT-style models.

PATMAT with AF-LDM-style resampling. The strongest shift-stability competitor is PATMAT with ideal Fourier resampling and an equivariance loss added, but without our attention or detail-branch changes. This is the "just transplant AF-LDM's recipe" baseline. Its reported encoder and latent SPSNR (45.10 and 40.94 dB, against 25.07 and 22.87 dB for the standard Stable Diffusion autoencoder and LDM) show how much stability the recipe buys in a different setting.

Ablations. Two ablations isolate our ingredients:

  • Full model minus the detail branch: tests whether the branch is what recovers identity.
  • Full model with windows restored: tests whether grid-free attention is what removes integer-shift drift.

Fairness. All variants share the same training data, mask distribution, resolution, iteration count, and hyperparameter search budget. We report training wall-clock time, because the Alias-Free ViT authors found their filtering raised training time from 69 to 487 hours.

What counts as beating the baselines. Beating PATMAT means at least halving the identity flip rate and the Shift-ID drift at non-window-multiple and fractional shifts. ArcFace similarity to reference must stay within 0.02, and FID within one point. Beating the AF-LDM-style baseline means matching its stability while having higher identity similarity to reference.

Why not just…?

Why not just train with random shift augmentation? It is the cheapest option, and for classification Azulay & Weiss found that augmentation leaves large residual sensitivity, because networks learn typical positions rather than invariance. Generation is untested, though, so we treat augmentation as a real threat. PATMAT is fine-tuned with jitter up to 16 pixels and evaluated at the same budget. If it matches our stability, the architectural changes are unjustified, and we will say so.

Why not just align and crop faces with a landmark detector first? Alignment removes most framing variation when the face is visible. But in large-hole inpainting the landmarks are often inside the hole. Manfredi et al. also showed that detectors themselves jitter under one-pixel shifts, so the canonical crop moves. We test this directly by running a detector on the masked images, aligning, inpainting, and measuring drift. Masks that cover few landmarks are the regime where alignment should win.

Why not just drop in BlurPool, or swap in an alias-free autoencoder? This is the most serious alternative, and it is exactly the AF-LDM-style baseline above. AF-LDM itself reports that filters alone are insufficient without an equivariance loss. Zhang notes that blurring costs fine detail in generation. Whether that detail cost actually shows up as an identity cost is the open question. If the baseline keeps ArcFace similarity at PATMAT's level, the detail branch is unnecessary and the contribution reduces to diagnosis plus grid-free attention.

Proposed Experiments

Data. Training and evaluation use CelebA-HQ at 512 pixels with MAT's large-hole masks. Celeb-ID (ExGANs) supplies at least three photos per subject, used for reference exemplars and held-out identity scoring. CelebAHQ-IDI is used for the PVA comparison, and PATMAT's seven-person set serves as a small out-of-distribution check.

Decisive checks. Three results settle the main questions.

  1. Window-grid test. MAT and PATMAT drift should be near zero at window-multiple shifts and large otherwise, and the criss-cross variant should erase that gap. If no gap appears, windows are not the cause and we redirect effort to aliasing and padding.
  2. Identity-cost curve. We sweep low-pass cutoff strength in the coarse path, with and without the detail branch, and plot ArcFace similarity to reference against Shift-ID consistency. The proposal succeeds if the detail branch moves the whole curve outward, giving more stability at equal identity. It fails if the two curves coincide.
  3. Border test. We place the hole within 32 pixels of the canvas edge versus centered. Padding-free borders should flatten the difference.

Reporting. Each result is reported with bootstrap confidence intervals over at least 2,000 image–mask pairs and three training seeds.