Boundary-Stopped Global Context for All-in-One Image Restoration

Restoration networks that remove haze, rain and noise with one set of weights need wide context. Haze thickness and rain streak patterns are only recognizable over large regions. The cheap ways to get that context, such as long horizontal and vertical sweeps, pooled strips and state-space scans, all average information across the image. That averaging has a side effect that restoration benchmarks rarely measure: at a faint boundary, where a polyp meets similar-looking tissue or a foggy car meets a foggy road, context from both sides leaks across, and the boundary comes out softer than it went in. The problem is worst where the image matters most for downstream use. A segmentation model reading the restored image then sees a smeared or invented contour. Ultra documented this failure for adverse-weather driving, where restoration hallucinations mislead segmentation.

We propose to let long-range context flow freely inside regions and stop at real object boundaries. A single learned 2D "boundary map" sets how much information survives each step of a linear-cost row and column sweep. The central difficulty is that the map must tell a real boundary from a rain streak or a noise edge. A multi-degradation training set provides a signal for this at no extra labeling cost. When the same clean image is corrupted in different ways, real boundaries stay where they are and corruption edges move or vanish. We therefore train the boundary map to agree across differently corrupted copies of one scene. The claim is that this map yields better restoration of low-contrast boundaries and better downstream segmentation from a single all-in-one model, and that it carries over to degradations never seen in training.

Core Claim

In all-in-one restoration, long-range aggregation should be controlled by an explicit boundary map that is shared across channels and directions. That map should be trained to be invariant to which degradation was applied. This design preserves low-contrast object boundaries that ungated global operators blur. It also keeps its advantage on unseen degradations, because the invariance objective ties the map to scene structure rather than to corruption statistics. The measurable prediction has three parts. At matched compute, the gated model should equal the strongest all-in-one baselines on overall restoration fidelity. It should beat them on fidelity inside thin bands around true boundaries. And it should beat them on the accuracy of a frozen boundary-sensitive segmenter applied to the restored output.

Where This Sits

Three strands of prior work supply the pieces, and none of them combines them. The first strand builds all-in-one restorers. AirNet learns a degradation code contrastively and adapts its receptive field with deformable convolution, but that adaptation is local. PromptIR modulates a Restormer-style decoder with learned prompts and is the strongest reported tri-degradation model in the file. MS-Mamba combines global and windowed state-space scans with a gradient block and a Fourier block. MS-Mamba is the closest prior work. It already recognizes that global scans blur edges, but its answer is to add a separate edge-restoring branch after the fact rather than to stop the scan from crossing edges. It evaluates only on PSNR/SSIM. MCLIR adds a task-agnostic contrastive loss in global VGG feature space. That loss does not specifically penalize boundary blurring, and it could be combined with our method.

The second strand supplies the mechanism. CCNet showed that two rounds of row-and-column attention give every pixel a full-image receptive field, using 11x less memory than dense non-local attention. Strip Attention (SANet) brought decoupled horizontal and vertical strips into restoration. Its strips have a fixed length $K$ that is the same everywhere in the image, and the file lists this rigidity as an open problem. Our gated sweep is a content-adaptive version of SANet's strip: the effective length at each pixel is however far the sweep travels before it reaches a boundary. The sequential multi-dimensional sweep itself goes back to Multi-Dimensional Recurrent Neural Networks.

The third strand motivates the evaluation. PraNet handles faint polyp boundaries with reverse attention. It reaches mean Dice 0.898 on Kvasir but only 0.628 on the unseen ETIS set, which shows how fragile boundary delineation is under small appearance shifts. It also assumes clean input. Boundary-focused methods such as the boundary-attention polyp network share that assumption, which is a constraint on our design: an edge detector run naively on a degraded image will fire on rain and noise. That is why our boundary map needs the invariance training. Ultra is the only work in the file that joins restoration and segmentation. It works at the level of task gradients, using Nash bargaining and causal intervention during training, and it is tied to specific segmentation backbones. We change the restorer's internal routing instead, and we require no segmentation labels or segmentation decoder at training time.

Mechanism

Within one image row, write $x_i$ for the feature vector at column $i$, $h_i$ for the running context, and $b_i \in [0,1]$ for the predicted boundary probability at that pixel. The forward horizontal sweep is

$$h_i = (1 - b_i)\, h_{i-1} + x_i ,$$

so context passes freely through pixels where $b_i \approx 0$ and is cut off where $b_i \approx 1$. We run this in four directions and then repeat the pass, as CCNet does, so that any pixel can reach any other pixel through a path that crosses no boundary. The cost is linear in the number of pixels. The map $b$ comes from a small convolutional head at each scale of a UNet. It is one channel, shared by all feature channels and all four directions. This sharing is deliberate: it makes the map interpretable and possible to regularize. A Mamba scan, by contrast, learns a separate decay for every channel and direction.

For training, we take a clean image $y$ and two sampled degradations $d_1, d_2$ drawn from noise, rain and haze. The objective is

$$\mathcal{L} = \sum_{k=1,2} \lVert R(d_k(y)) - y \rVert_1 + \lambda\, \lVert b(d_1(y)) - b(d_2(y)) \rVert_1 ,$$

where $R$ is the restorer and $\lambda$ weights the agreement term. One trivial solution satisfies the agreement term: $b \equiv 0$ everywhere, which means ungated global averaging. Only the reconstruction loss discourages it. We will therefore report the mean and sparsity of $b$ throughout training, and we treat collapse as a real risk rather than dismissing it. Constructing paired corruptions of the same image is straightforward for synthetic noise and rain streaks. For haze it requires depth, which RESIDE's indoor training set provides.

Research Program

We first build the gated sweep block and place it inside a PromptIR-sized four-level UNet. The block replaces the Transformer attention, and the prompt modules are kept, so the only change is how long-range context is routed. We then train in the standard three-degradation all-in-one setting, adding paired corruptions for the agreement term.

The analysis asks three questions. First, does the boundary map line up with boundaries in the clean image and ignore corruption edges? We measure this as the correlation between $b$ and edges in the clean image, compared with its correlation to a residual map (degraded input minus clean). Second, does that alignment survive on degradations excluded from training, namely low light, snow and blur? Third, does any restoration gain concentrate in boundary bands, as the claim predicts, rather than being spread uniformly over the image?

Baselines

The naive reference is to feed the degraded image, and separately the output of an ungated version of our own network, straight into the downstream evaluator. This shows how much any restoration helps boundaries at all.

The strongest existing competitor is PromptIR. It is the best reported all-in-one model in the file, beating AirNet by an average of 0.86 dB across the tri-degradation tasks. The file does not give PromptIR's individual Rain100L score, but AirNet's reported Rain100L PSNR is 34.90 dB. MS-Mamba is the second strong competitor because it targets the same trade-off between global context and edge preservation, and the file has no reported all-in-one number for it. SANet is retrained in the all-in-one setting; its reported single-task result is 26.29 dB on DPDD. All three are retrained on our exact data mixture and schedule. Where this is feasible they are matched to within 10% in parameters and FLOPs, and each gets the same hyperparameter search budget. If a model cannot be matched, we report its native size and a scaled-down version.

Winning means two things. The first is overall PSNR within about 0.2 dB of PromptIR on Rain100L, SOTS and BSD68. The second is a clear improvement in PSNR within the boundary bands and in downstream Dice, holding across three training seeds.

Four ablations isolate the key ingredient. The first uses a constant gate, which gives ungated sweeps and tests whether stopping at boundaries matters. The second learns the gate without the agreement term, which tests whether invariance matters. The third supervises the gate directly with Sobel edges of the clean target, which tests whether clean-edge supervision is enough. The fourth adds a fixed edge loss to PromptIR.

Why not just…?

Why not just add an edge loss to PromptIR? This is the cheapest competitor, and Ultra's observation that fixed-weight combinations of objectives produce conflicting gradients is only suggestive here, because Ultra studied restoration versus segmentation rather than restoration versus edge losses. An edge loss rewards sharp output edges, but it does not stop the network from averaging across a faint boundary in its intermediate features. It may also sharpen hallucinated edges. We run this baseline, with $\lambda$ tuned, on the same budget. If it matches our method in boundary-band PSNR and in downstream Dice, the routing idea adds nothing, and we will say so.

Why not just use MS-Mamba, whose selective scan already learns input-dependent decay? This is the most serious threat. A per-channel, input-dependent decay could in principle learn to stop at boundaries. Our bet is that, without an invariance constraint, it will just as readily learn to stop at rain streaks and noise, because nothing in its objective tells the two apart. We test this directly. We extract MS-Mamba's effective decay maps and compute the same two correlations we use for our map, one against clean-image edges and one against the corruption residual. We also add our agreement term to a Mamba scan as a hybrid. If the hybrid does as well as our shared single-channel gate, the contribution is the invariance objective rather than the specific gate design, and we will present it that way.

Why not just supervise the gate with edges from the clean training image? Synthetic training pairs contain the clean image, so this supervision costs nothing. It may work as well as the agreement term on the degradations seen in training. There are two reasons to expect it to fall short. Clean-image Sobel edges respond weakly at exactly the low-contrast boundaries we care about, and they fire on texture. And the agreement objective can also be applied to real unpaired images, where no clean target exists. The clean-edge ablation, evaluated on unseen degradations and on boundary bands around polyp masks, settles the question. If it ties, we adopt the simpler supervision.

Why not just restore first and then refine boundaries with a segmenter such as PraNet? This addresses a different question, because it improves the segmenter rather than the restorer. PraNet's drop from 0.898 to 0.628 Dice under a modest domain shift, and Ultra's hallucination loop, both suggest that restorer artifacts do carry into downstream boundaries. Our downstream protocol measures exactly this pipeline, with the segmenter frozen, so any difference between restorers comes from how well each one preserves boundaries.

Proposed Experiments

Restoration is evaluated on the standard all-in-one splits: Rain100L for deraining, SOTS for dehazing, and BSD68 and Urban100 at noise levels $\sigma$ of 15, 25 and 50 for denoising. For each we report whole-image PSNR/SSIM and PSNR inside a 5-pixel band around edges of the clean image. Cross-task generalization is tested on degradations absent from training: LOL-v2 for low light, CSD for snow, and GoPro and DPDD for blur. We also test PromptIR's protocol of training on $\sigma=50$ and testing on $\sigma=100$, where PromptIR reports a lead of about 7 dB over AirNet.

For low-contrast boundaries, we apply held-out synthetic noise, haze and low light to Kvasir and ETIS. We restore each image with every model and score it with a PraNet trained on clean Kvasir and then frozen. The clean-input references are PraNet's reported Dice of 0.898 on Kvasir and 0.628 on ETIS. We report Dice, plus boundary F-score computed on the polyp mask contours. On real weather, we restore ACDC images and score them with a frozen off-the-shelf segmenter, reporting mIoU and boundary F-score. Ultra's 73.0% mIoU with HRDA serves only as context, because Ultra adapts the segmenter and we do not.

There are three decisive checks. The first is whether the benefit concentrates in boundary bands and in downstream Dice, with no loss in overall PSNR, relative to PromptIR and MS-Mamba at matched compute. The second is whether our boundary map aligns with clean-image edges and not with the corruption residual on unseen degradations, while the ungated and unconstrained variants do not show this separation. The third is whether the agreement-trained gate beats the clean-edge-supervised gate specifically on unseen degradations and on low-contrast polyp boundaries. If the first check fails, the idea is wrong. If only the third check fails, the method becomes simpler, with the gate supervised by clean-image edges, and the gated routing remains the finding.