Spending Native-Resolution Compute Where Viewers Notice Facial Errors

People forgive blur and noise in the background of a portrait but reject a photo when an eyelid, iris, or lip contour is slightly wrong. Yet restoration models that work on large images either process every region the same way or crop the face out and handle it separately. This proposal makes that human asymmetry into a measured quantity and uses it to decide where an image model gets its expensive full-resolution processing. We will first measure, with a small perceptual study, how detectable localized distortions are across facial regions, face sizes, and head poses. We then train a 4K portrait super-resolution model that runs a cheap whole-image pass and adds full-resolution correction tiles only where the measured sensitivity is high and the cheap pass is likely to be wrong.

The bet is that a portrait restorer given a fixed, small number of full-resolution tiles placed by human sensitivity will match the facial quality of a model that tiles the whole canvas, at a fraction of the cost. It should also beat a model that places tiles using only a binary "face here" mask, and it should avoid the jawline seams of crop-and-paste pipelines. If the binary mask does as well as the measured sensitivity map, the perceptual-measurement half of the project has no value, and the experiments are built to show that clearly.

What exists and what is missing

Two pieces of evidence motivate the asymmetry. HeadsUp! High-Fidelity Portrait Image Super-Resolution observes that the face covers about 20% of a portrait but drives about 80% of judged quality. Seeing is not always believing finds that people detect AI-generated images more accurately when the subject is a face, and that 28% of the cues they report are detail problems. Earlier restoration work adds a constraint on design: observers prefer a slightly smooth face to a sharp one with anatomical errors (Schölkopf et al.), and a sharp background does not compensate for a distorted face (Li et al.). Taken together, extra detail in the wrong place, or wrong detail on a face, is a cost rather than a benefit.

The method borrows from two families.

Whole-image portrait restoration. HeadsUp trains a single-step latent diffusion model on the whole image. A latent diffusion model is a generator that denoises in a compressed feature space roughly 8× smaller than the image. HeadsUp adds a face-region loss and an identity loss. This removes seams, but the face is still reconstructed through the compressed latent, so structures smaller than a latent cell, such as iris rims and eyelid edges, have no full-resolution path.

Native-resolution tiling. No Pixel Left Behind (HiDA-Net) shows, for fake-image detection, that adding native-resolution tiles to a resized global view keeps high-frequency evidence that resizing destroys. Its tiles are spread uniformly, and the paper names that compute overhead as a limitation.

Component-level face restoration. DFDNet and DGFormer fix specific regions, but only four components (eyes, nose, mouth) on tight crops located by landmark detectors. Those detectors fail under heavy degradation, and the methods ignore jawline, hairline, and neck boundaries. FaceFormer and SGPN likewise operate only on cropped faces.

None of these decides where full-resolution capacity goes using measured human sensitivity. None of them treats face-to-background boundaries as regions that need it.

Core Claim

Viewer sensitivity to local distortion varies strongly and predictably within and around the face, and the variation is not captured by a binary face mask. Examples are iris and eyelid versus cheek, frontal versus profile, and a large face versus a small one in frame. A sensitivity map predicted from the low-resolution global view, combined with an estimate of where the global pass errs, is a better rule for allocating a fixed budget of full-resolution tiles than uniform tiling, a face mask, or landmark-based components.

Research Program

Part 1: Measure sensitivity. We will insert controlled local distortions into clean high-resolution portraits. The distortions are small geometric warps, blur, and swaps of hallucinated texture. They are placed in about a dozen region types: iris, eyelid, brow, nostril, lip contour, cheek skin, jawline, hairline, ear, neck, hands, and background texture. We vary face size in frame and yaw angle, including profiles, because prior-based methods are known to fail on profiles.

Participants do two-alternative forced-choice trials, picking which of two images is altered. A staircase procedure, which adjusts distortion strength trial by trial, estimates the detection threshold for each condition. We then fit a small network that predicts a per-pixel sensitivity map $s(x)$ from a downsampled image, using face parsing and pose only as training targets. At test time the map comes from the global view alone, so it does not depend on landmark detection surviving the degradation.

As a cheap cross-check, we ask whether the gradients of the face quality model from Going the Extra Mile in Face IQA, which is trained on GFIQA-20k opinion scores, reproduce the measured map. If they do, later work could skip human studies.

Part 2: Budgeted two-scale restorer. The global stream is a single-step latent diffusion super-resolver of the HeadsUp/OSEDiff type, applied to the whole canvas. A lightweight pixel-space refinement network then runs on $K$ native-resolution tiles. It reads global features through cross-attention and outputs a residual correction. Each tile's residual is tapered to zero at the tile border and trained with an overlap-consistency loss, so tiles cannot create seams.

Tiles are chosen to maximize expected perceptual gain:

$$T^* = \arg\max_{|T| = K} \sum_{t \in T} \bar{s}_t \, \hat{e}_t,$$

where $T$ is the set of chosen tiles, $\bar{s}_t$ is the mean predicted sensitivity in tile $t$, and $\hat{e}_t$ is a predicted reconstruction error of the global pass for that tile, from a small error-prediction head. The training loss on tile outputs is weighted by $s(x)$.

The product form matters for two reasons. A sensitive region that the global pass already renders well should not consume budget, and a badly rendered patch of background wall should not either.

Baselines

The naive reference is bicubic upsampling plus the generic uniform super-resolver OSEDiff. HeadsUp reports PSNR 25.19 dB and SSIM 0.7802 for OSEDiff on PortraitSR-4K. It shows how much face awareness buys at all.

The strongest existing method is HeadsUp, which reports PSNR 25.64 dB and SSIM 0.8060 on PortraitSR-4K. It is the strongest competitor because it already solves the seam problem and already up-weights the face in its loss. Our global stream uses the same backbone family and LoRA fine-tuning recipe. If HeadsUp code is unavailable, we reimplement its face-region and identity loss on that shared backbone and also report against its published numbers on the same test split. Beating it means two things. First, a statistically significant human preference in paired comparisons restricted to face crops and to jawline/hairline crops. Second, better face-crop identity similarity and lower face-crop LPIPS, at no worse whole-image PSNR than 25.64 dB.

The crop-restore-paste pipeline pairs CodeFormer on the face with Real-ESRGAN on the background, blended by feathering. HeadsUp describes its seams but gives no number for it; no reported result found on Lacuna for this pipeline on PortraitSR-4K. It is the most common practical tool, so it is the competitor to beat on boundary artifacts.

Uniform tiling is a HiDA-Net-style setup that applies our refinement network to all tiles. It is run both at matched compute (same $K$, tiles chosen at random) and unmatched (every tile). This is the upper bound our allocation should approach.

Three ablations isolate the key ingredient by replacing $\bar{s}_t$ with each of the following:

  1. a binary face-parsing mask;
  2. landmark-detected component boxes in the style of DFDNet;
  3. the constant 1, which leaves error-only allocation.

All variants share data, backbone, training steps, and hyperparameter search budget. Compute is reported as FLOPs and latency at 4K.

Why not just…?

Why not just weight the loss by a face mask, as HeadsUp does? Weighting the loss changes what the model tries to fix but not what it can resolve. Sub-latent-cell facial structure still passes through an 8× compressed representation. The comparison against HeadsUp tests whether a full-resolution path adds anything beyond that. If HeadsUp matches our face-crop human preference, the second scale is unnecessary and we will report that.

Why not just crop the face, restore it with a specialist, and paste it back? HeadsUp documents seams, lighting steps, and jawline boxes from such pipelines. Our psychophysics directly measures whether jawline and hairline errors are as detectable as we expect. Our boundary-crop human study then tests whether tapered residual tiles avoid these errors where pasting does not.

Why not just tile everything at native resolution? It probably works, and it is the quality ceiling. It is also the cost problem that HiDA-Net and Thinking in Scales flag. The claim is only that sensitivity-guided placement reaches most of that ceiling at much lower $K$. We will plot face-crop quality against $K$ for each allocation rule.

Why not just allocate tiles to the face mask? This is the real threat. On frontal, mid-sized faces a mask may be nearly as good. The proposal survives only if the measured map wins where it should differ from a mask: profiles, small faces, boundary regions, and heavy degradation where parsing fails. These are exactly the slices we evaluate. If it does not win there, we drop the claim that measured sensitivity matters.

Proposed Experiments

Main evaluation. Use PortraitSR-4K and report PSNR, SSIM, face-crop LPIPS, face-crop identity similarity, and FLOPs. Also report the score of the GFIQA-trained face quality model; that paper reports PLCC 0.9652 for it on GFIQA-20k. Run paired human comparisons on face, boundary, and background crops.

Robustness slices. WIDER-Test probes heavy in-the-wild degradation and pose. DAEFR reports FID 36.720 there on cropped faces, which we cite only as context because our setting is whole-portrait. EDFace-Celeb-150K splits results by degradation type, so we can see where landmark-based allocation collapses and where the global-view map holds.

Decisive checks. There are three:

  1. Measured detection thresholds must differ across face regions by more than they differ between face and background on average. If they do not, a binary mask is sufficient.
  2. At equal $K$, sensitivity × error allocation must beat mask allocation on the profile and small-face slices.
  3. Boundary-crop preference must favor our model over crop-restore-paste.

Failing check 1 or check 2 would reduce the project to HeadsUp plus a refinement head, and we would report it that way.