Letting the Codebook Decide Where to Trust the Input: Per-Location Bypass Gating for Face Restoration

Blind face restoration takes a badly degraded photo (blurred, noisy, compressed, low resolution) and tries to produce a sharp version of the same person. A strong recent approach restricts the output to a fixed dictionary of clean image pieces, called a codebook, learned from high-quality faces. The degraded photo is encoded, a Transformer predicts which dictionary entry belongs at each location, and a decoder paints the result. Used alone, the dictionary produces clean faces that drift away from the real person. So systems also pass the original encoder features straight to the decoder, a bypass or skip connection, and blend the two with a single hand-set dial. That dial is the weak point. One number applies to the whole image, so turning it up to keep identity also carries blur, noise and occluder artifacts through the bypass, and turning it down to remove them discards identity everywhere.

This note proposes making that dial vary by location, and computing it from the codebook itself rather than from a separately trained degradation detector. At each location we measure how far the encoder's feature sits from the dictionary entry the Transformer predicted there. Where the two roughly agree, the content is a degraded face that the dictionary can explain, so the output should trust the dictionary. Where they strongly disagree, the content is something the dictionary cannot represent, such as a hand, a microphone, or unusual eyeglasses, so the output should trust the input. A second, smaller problem follows. The dictionary lives on a coarse fixed grid, while the bypass carries features at higher resolution. Stretching coarse-grid decisions up to fine resolution recreates the rigid-partition failures documented in person re-identification. We therefore borrow a soft reassignment step from that literature so that gate boundaries follow object edges instead of grid lines.

Core Claim

The central bet is that the distance between an encoder feature and its context-predicted code separates two kinds of location that call for opposite treatment. Degraded face regions should mostly follow the codebook. Non-face or out-of-dictionary content should mostly follow the input. If that separation holds, a gate built from this distance should improve on a global blending weight on real, heavily degraded and partly occluded faces. It should also generalize better than a gate trained freely on synthetic degradations, because the signal comes from the clean-face prior rather than from the specific corruptions seen during training.

What Prior Work Establishes and Leaves Open

CodeFormer supplies the method we build on. It learns a face codebook on FFHQ, predicts code indices with a global Transformer, and blends encoder features back in through a Controllable Feature Transformation module whose strength is set by one scalar $w$ for the whole image. The paper itself notes that low $w$ favours quality and high $w$ favours fidelity. It also reports failures on unaligned faces and on accessories outside the dictionary. We keep its codebook and Transformer unchanged and replace only the scalar $w$.

Three lines of work motivate filtering the bypass, but none of them does so under a discrete prior. UCTransNet shows that raw skip connections can make a U-Net worse than having no skips at all on GlaS, and replaces them with channel-wise cross-attention. TSDN shows that skip connections let defects bypass the bottleneck: adding them drops image-level AUC from 87.3% to 70.7% in their ablation. It then gates skip channels with a learned normality estimator. FreeU rescales skip features in the frequency domain after training, using global constants. All three operate on continuous features, and each learns its filter or fixes it by hand. None uses a clean-data dictionary to decide what to block.

The closest mechanism is ORFormer. For landmark detection, it flags occluded patches by the cosine disparity between each patch token and a context-aggregating "messenger" token, then replaces the flagged patches. Our disagreement signal is the same kind of object. The difference is where it is applied. ORFormer uses it to fix sparse landmark heatmaps. We use it to control how much of the input flows through the bypass of a pixel-level restoration decoder, and we compare the input against a discrete clean code rather than a continuous context token.

The rigid-partition literature constrains how the gate is placed onto high-resolution bypass features. PCB with Refined Part Pooling shows that fixed stripes are a strong starting point, and that softly reassigning each column feature to the part it most resembles fixes misalignment. AAformer replaces stripes with optimal-transport part tokens, but its equal-partition constraint can force clutter into parts. That is a reason not to adopt it for gating, where occluders may cover a small or a large area.

Research Program

Let $\hat{z}_i$ be the encoder feature at codebook-grid cell $i$, and let $c_i$ be the codebook vector the Transformer predicts there. The disagreement score is

$$d_i = \frac{\|\hat{z}_i - c_i\|_2}{\mu_{r(i)}},$$

where $\mu_{r(i)}$ is the mean of $\|\hat{z}-c\|_2$ over degraded training faces in the same face region $r(i)$. The normalization reflects that hair and skin naturally sit at different distances from their codes. The per-cell bypass weight is $w_i = \sigma(\alpha(d_i - \tau))$. Here $\sigma$ is the logistic function, and $\alpha$ and $\tau$ are two scalars fit on a small validation set. The gate therefore has almost no learned parameters, and this is deliberate: it is what makes the comparison with a freely learned gate informative. A user-level global dial is kept by shifting $\tau$, so fidelity stays controllable.

To transfer $w_i$ to a higher-resolution bypass layer, each fine-resolution feature is softly assigned to neighbouring coarse cells according to feature similarity, in the manner of Refined Part Pooling. It then inherits their gates in proportion. The alternative is plain bilinear upsampling. Both are implemented, and the difference between them is measured specifically at occluder boundaries.

The first work package is diagnostic and precedes any restoration numbers. On CelebA-Test faces with synthetic degradation and pasted occluders (hands, microphones, and eyeglasses taken from other images), we measure how well $d_i$ separates occluder cells from degraded-face cells, reported as area under the ROC curve at each degradation severity. The obvious risk is that severe blur pushes face features as far from their codes as an occluder is. If separation collapses at high severity, the approach fails in exactly the regime that matters, and we would learn that before building anything further.

Baselines

The naive reference is CodeFormer with its global $w$, run at the value its authors recommend. It is also swept per test image with oracle selection of the best $w$. The oracle sweep is the honest ceiling for any global dial. Beating it at a fixed setting would show that per-location control adds something no single scalar can match.

The strongest competitor is a learned per-location gate: a small convolutional head that takes encoder and decoder features and outputs $w_i$. It is trained end-to-end on the same FFHQ synthetic-degradation data, with the same occluder augmentation, the same schedule and the same tuning budget. This isolates the key ingredient, because it has the same gate resolution and differs only in whether the gating signal comes from the codebook or is learned freely. A TSDN-style normality estimator, adapted to supervise the gate with the known occluder masks from augmentation, forms a second learned variant.

A frequency-domain baseline applies FreeU-style scaling to the bypass, with the scaling constants tuned on validation data. Two ablations complete the set. The first swaps the context-predicted code $c_i$ for the nearest-neighbour code, to test whether Transformer context matters. The second swaps soft reassignment for bilinear upsampling, to test the partition question.

The file contains no reported LPIPS or identity-score value for CodeFormer on CelebA-Test, WIDER-Test, LFW-Test or WebPhoto-Test (no reported result found on Lacuna for these). All baselines will therefore be rerun from released weights in one evaluation harness. The proposal counts as beating the baselines only if it improves identity similarity on occluded and out-of-dictionary subsets without worsening LPIPS on unoccluded CelebA-Test, compared against both the oracle global $w$ and the learned gate.

Why not just…?

Why not just learn the spatial gate end-to-end? This is the real threat, and it may match us on synthetic test data. The argument for the codebook-derived signal concerns shift. A learned gate sees only the degradations and occluders in its training augmentation. The disagreement score measures departure from a clean-face prior, so it should transfer to real corruptions. The decisive test trains both gates on synthetic data and evaluates them on WIDER-Test, on occluder types held out of the augmentation, and on real occluded faces from COFW. If the learned gate matches us there too, the contribution shrinks to a cheaper, nearly parameter-free gate, and we will report it that way.

Why not just tune the global weight per image? The oracle-sweep baseline answers this directly. A single scalar cannot keep a hand sharp while also cleaning a blurred cheek in the same image. If the oracle global $w$ comes within noise of our gate on the occluded subset, the problem is not spatial and the idea is wrong.

Why not just mask occluders with a face parser? Parsers are separate models that are fragile under exactly the heavy degradation targeted here, as the re-identification literature notes when motivating end-to-end part discovery (AAformer, SCFP). They also label accessories inconsistently. We include a parser-mask gate as a secondary baseline on WIDER-Test, reporting both its accuracy and its added runtime against CodeFormer's reported 0.07 s per 512×512 image on a V100.

Why not just filter the bypass in frequency, as FreeU does? FreeU's constants act identically on occluded and clean regions (FreeU). It has no way to preserve a hand while suppressing blur on the adjacent cheek. The occluded-subset comparison will show whether this matters in practice.

Proposed Experiments

Training uses FFHQ with CodeFormer's degradation pipeline, extended by pasted occluders. Evaluation covers CelebA-Test (LPIPS, PSNR, and identity similarity from a face-recognition embedding), an occluded CelebA-Test variant built with known masks, and a subset containing eyeglasses and other accessories. We also evaluate on LFW-Test, WebPhoto-Test and WIDER-Test, using no-reference quality and identity consistency across restorations.

As a downstream check of structural faithfulness, we degrade COFW images, restore them, and run a landmark detector on the result. ORFormer's reported 4.46 NME on clean COFW (ORFormer) serves as a reference, since it is the published figure on undegraded images. Lower NME after restoration indicates the gate kept real geometry rather than hallucinating a canonical face.

Three results decide the question. First, the diagnostic ROC must show usable separation between occluder cells and degraded-face cells at high severity. Second, the gate must beat the oracle global $w$ on occluded and accessory subsets while holding LPIPS on clean-face regions. Third, the gate must beat or match the learned gate out of distribution, on WIDER-Test and held-out occluders, where the argument for a prior-derived signal lives. The soft-reassignment ablation is reported separately as boundary error along occluder masks, since that is where rigid grid-cell gating should visibly fail.