Running an image restoration model (a dehazer, a deraining network, a super-resolver) over a whole photo is wasteful when only part of it is degraded. It can also do harm: restorers distort regions that were already clean, and a downstream object detector pays for that distortion. The natural fix is to restore only the degraded patches and leave the rest untouched. That fix creates a new defect. Wherever restored pixels meet original pixels, there is a visible seam: a jump in color, contrast, or texture that neither the restorer nor the original image explains. This note proposes treating that seam as a small generation problem with known content on both sides. A thin band around the boundary of the restored region is masked out and refilled by a bidirectional masked token model in the style of MaskGIT. Such a model fills every masked position in parallel over a handful of rounds, and in each round it commits only its most confident guesses.
The reason to use this kind of generator rather than a blending rule or a regression network is specific. A seam has two plausible appearances, the restored one and the original one. A method that predicts each pixel independently, or regresses to their mean, produces the visual equivalent of "thank thanks": ghosted, blurred, half-and-half texture. Iterative confident-first decoding was built to avoid exactly this kind of blending in parallel generation. A thin band also keeps the model cheap enough that selective enhancement still saves compute overall.
Core Claim
When a strong generative restorer, one that synthesizes new high-frequency texture, is applied only to selected patches, the boundary band can be repaired by a conditional masked token model in roughly 8 parallel steps. The repair should make seams statistically indistinguishable from patch interiors, preserve high-frequency texture where regression and blending blur it, and cost a small fraction of applying overlapping-patch diffusion over the selected region. We do not claim this for mild, deterministic restorers. There, a small regression network may be enough, and the experiments are designed to show where that line falls.
Where This Sits
Three bodies of work frame the proposal.
The case for selective enhancement comes from Kumar et al. and From Fog to Failure: How Dehazing Can Harm Clear Image Object Detection. Both report that uniform enhancement wastes compute and corrupts clean regions, and they restrict restoration to detected degraded regions. Neither says how the restored region should be joined back to the untouched image, and that join is our target.
Selective computation for recognition is well developed, but it either ignores or deliberately avoids generating pixels at boundaries. Patch Proposal Network for Fast Semantic Segmentation of High-Resolution Images routes low-confidence patches to a refinement branch and fuses features without modeling the interface between refined and unrefined patches. Sparse Refinement for Efficient High-Resolution Semantic Segmentation refines only high-entropy pixels. Look Closer To Segment Better: Boundary Patch Refinement for Instance Segmentation refines crops along mask boundaries but stitches them back naively, which the file identifies as the "seam shifting" problem. All three predict labels, not appearance, so the two-sided ambiguity we target does not arise for them.
Seam handling in generative models supplies both the method and the main constraints. Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models removes tile seams by averaging noise estimates across overlapping patches at every reverse step. It also reports that image-space blending blurs, which motivates our rejection of blending. Its cost is the constraint: it is dense and non-selective, and takes about 20 seconds for a 640×432 frame. Tiled Diffusion: Seamless Multi-Image Generation via Latent Constraints enforces continuity by padding each tile's latent with slices from its neighbors, and it also runs uniformly over the whole grid. MaskGIT: Masked Generative Image Transformer supplies the decoding procedure: a bidirectional transformer over vector-quantized tokens, a cosine masking schedule, and confidence-ordered unmasking.
MaskGIT also reports two failure modes that bear directly on this design. The first is structural drift in outpainting. Our band is bounded on both sides by known content, so the task is inpainting a narrow strip rather than extrapolating, which should sidestep drift. The second is quantization artifacts from the tokenizer. This is a real risk, and the program addresses it explicitly. Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement and Promises and Pitfalls of Generative Masked Language Modeling supply the explanation of why single-pass parallel prediction blends modes and why iterative refinement helps. That explanation is the hypothesis our ablations test in the image domain.
Research Program
The pipeline has three parts. A lightweight selector marks degraded patches, giving a region $S$. A heavy restorer $f$, which is interchangeable, processes only the patches in $S$. A seam model then repairs the band. Let $y$ be the degraded input and $B$ the set of pixels within distance $r$ of the boundary of $S$. The composite image is $c = f(y)$ on $S \setminus B$ and $c = y$ outside $S \cup B$, with $B$ left empty.
The seam model tokenizes a window around $B$ with a VQGAN-style tokenizer. It keeps tokens outside $B$ fixed, masks the tokens inside $B$, and additionally conditions on the degraded pixels $y$ inside $B$ so that it knows what content lives there. Over $K$ steps it predicts all masked tokens in parallel and unmasks the most confident fraction according to MaskGIT's cosine schedule. Training uses synthetic local degradations applied to clean images $x$, so ground truth exists everywhere. The objective is MaskGIT's masked-token cross-entropy toward the tokens of $x$ inside $B$:
$$\mathcal{L} = -\mathbb{E}\Big[\sum_{i \in M} \log p_\theta\big(z_i(x) \mid z_{\bar M}(c),\, y_B\big)\Big],$$
Here $z_i(x)$ is the token of the clean image at position $i$, $M$ is a random subset of band positions masked under the cosine schedule, $z_{\bar M}(c)$ are the visible tokens of the composite, and $y_B$ is the degraded input inside the band.
To make the model independent of any one restorer, the training composites are produced by several restorers. At test time we also evaluate restorers held out of training.
Tokenizer artifacts are the main technical risk. Pasting the decoded band back creates a secondary seam at the band's outer edge, where the tokenizer's reconstruction meets real pixels. We handle this with a small pixel-space decoder head. The head takes the predicted band tokens together with the real pixels just outside the band, and is trained with an L1 penalty plus a gradient-continuity penalty on the outer edge. Every evaluation measures seams at both the inner and outer edges of the band, so this failure cannot hide.
As a secondary question, we study whether the band model can absorb small selection errors. Because it conditions on $y_B$, it should be able to correct degraded pixels that the selector narrowly missed near $\partial S$. We test this by perturbing the selector's masks.
Baselines
The naive reference is hard pasting: the restorer's output inside $S$, the original pixels outside it, and no seam treatment. This fixes the size of the problem we are solving. We also report full-image restoration with the same restorer, which shows what selectivity gains in clean-region fidelity and downstream accuracy, as From Fog to Failure suggests it should.
The strongest quality competitor is selective overlapping-patch diffusion. This is the method of Özdenizci and Legenstein restricted to the patches covering $S$ plus a margin, with noise averaged across overlaps and the pixels outside $S$ clamped to the input at each step. It is the strongest competitor because it is the only existing method that removes seams inside a generative process rather than after it. We run it with the same restorer backbone where possible, the same selected region, and the same training data, and we tune its overlap stride and step count on the same validation split with the same tuning budget as our model. The paper reports roughly 20 seconds per 640×432 frame for the dense version. We found no reported result on Lacuna for seam quality or downstream accuracy under selective local degradation, so this baseline must be measured fresh.
Beating it means two things. Our seam metrics must be within its confidence interval or better. Our wall-clock time for seam repair must be at least an order of magnitude lower at matched selected area.
A regression baseline uses a small U-Net trained on exactly our data and band masks with L1 plus perceptual loss, at matched parameter count and training compute. It isolates whether generation is needed at all.
The key ablation runs our own model at $K=1$, meaning a single parallel pass with no confident-first ordering, against $K=8$. If the claimed coordination benefit exists, it must show up in this comparison.
For reference on seam metrics, Tiled Diffusion reports a Tiling Score of 0.03 against 0.29 for its baseline. That score is the mean absolute pixel difference across boundary offsets, measured on generated tiles rather than restoration composites. We adopt the metric, not the number.
Why not just…?
Why not just dilate the selected region and feather-blend? This is the cheapest fix, and it would win if restorers were close to the identity on clean pixels. The fog-to-failure evidence says they are not: dehazing visibly changes clear regions, so a mismatch remains inside the feathering ramp. Özdenizci and Legenstein report that blending in image space blurs. We include dilate-and-feather and Poisson gradient-domain blending as baselines. The band high-frequency energy ratio defined below directly measures the blur they are expected to introduce.
Why not just run selective overlapping-patch diffusion? On quality this is a real threat, and it may match or beat us. Our argument is cost. It needs hundreds of reverse steps over every patch touching $S$, while our model touches only the band in about 8 steps. The matched-area timing experiment settles the question. If diffusion's seam quality is clearly better and the gap cannot be closed with larger $K$, then our contribution shrinks to a speed–quality tradeoff point, and we will report it that way.
Why not just train a feed-forward U-Net to fix the band? For deterministic restorers whose outputs are smooth, this probably works as well as our model, and we expect to concede that regime. The hypothesis is that it fails when the restorer hallucinates texture, because regression toward two incompatible appearances averages them. The comparison of U-Net, $K=1$, and $K=8$, stratified by restorer type (regression restorer versus generative restorer), decides it. If the U-Net matches $K=8$ even with generative restorers, the core claim is false.
Proposed Experiments
The primary benchmark is built from Cityscapes images. We composite local, spatially varying haze, rain streaks, and blur onto random regions, which gives clean ground truth plus segmentation labels. We also build a VisDrone2018 variant with the same local degradations for small-object detection. VisDrone is the setting where The Power of Tiling for Small Object Detection showed that tile borders matter for small objects. Restorers include at least one regression network and one diffusion-based restorer.
Metrics fall into four groups. Seam quality uses the Tiling Score evaluated across the inner and outer band edges. Texture preservation uses the band high-frequency energy ratio: the band's high-pass energy divided by that of adjacent interior patches, which should equal 1 if there is no blur. Fidelity uses PSNR and LPIPS inside the band and over the whole image. Downstream impact uses SegFormer or HRNet segmentation mIoU and boundary F-score on Cityscapes, and detector mAP on VisDrone, all run on the repaired images.
To separate seam errors from selection errors, every comparison is run twice: once with oracle degradation masks and once with a learned selector.
The decisive checks are as follows. First, $K=8$ must beat $K=1$ and the U-Net on the band high-frequency ratio and LPIPS for generative restorers, while roughly tying them for regression restorers. That pattern would confirm the coordination explanation and not merely added capacity. Second, our method must reach seam scores within confidence intervals of selective overlapping diffusion at a tenfold or greater reduction in seam-repair time. Third, the seam at the band's outer edge must not exceed hard pasting's seam score. If it does, the tokenizer has simply moved the seam outward. Fourth, under selector masks perturbed by a few patches, the band model should recover part of the downstream accuracy lost to missed degradation.
We also test held-out restorers. If performance collapses on them, the model has learned one restorer's appearance rather than how to resolve a seam, and we would retrain with a broader set of restorers before drawing conclusions.
References
- From Fog to Failure: The Unintended Consequences of Dehazing on Object Detection in Clear Images
- Patch Proposal Network for Fast Semantic Segmentation of High-Resolution Images
- Sparse Refinement for Efficient High-Resolution Semantic Segmentation
- Look Closer To Segment Better: Boundary Patch Refinement for Instance Segmentation
- Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models
- Tiled Diffusion
- MaskGIT: Masked Generative Image Transformer
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
- Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
- The Power of Tiling for Small Object Detection