Restoring Images Without Merging Touching Objects

When an image restoration network removes blur, noise, or low resolution from a crowded scene, it tends to fill in the thin gap between two touching objects of the same kind. Two nuclei pressed together, two adjacent buildings, or two parked cars become one smooth blob, and the brightness of one object leaks into its neighbour. Methods that guide restoration with a semantic map, meaning a per-pixel label of what class is present, do not fix this. They tell the network "this is a cell" but not "this is where one cell ends and the next begins." We propose to guide restoration with a predicted instance boundary field instead: a per-pixel probability that the pixel lies on a contact line between two objects. The network uses this field to stop mixing information across those lines, and it re-estimates the field from its own partially restored output.

The practical stake is concrete. In microscopy and remote sensing, the restored image is often measured, not just viewed. Investigators read off per-cell fluorescence intensity or per-parcel spectral averages. If restoration bleeds signal across a contact, every downstream per-object measurement is corrupted, even if a separate segmenter later draws the right masks. The question we want to settle is whether restoration needs an instance-separation prior rather than a class prior, and whether that prior must act on how information flows inside the network rather than only on the inputs or the loss.

Core Claim

Class-conditioned restoration applies the same modulation to every pixel of a class. That makes it structurally biased toward texture and intensity continuity across same-class contacts. Our hypothesis is that replacing class conditioning with boundary-gated spatial mixing reduces cross-instance signal leakage and preserves contact gaps. In boundary-gated mixing, each pixel aggregates features mainly from neighbours that are not separated from it by a predicted contact line. We expect this to measurably beat three cheaper ways of injecting the same boundary information: input concatenation, feature modulation, and a boundary-weighted loss. We further claim that a single refinement pass is enough to break most of the "chicken-and-egg" problem, in which boundaries are needed to restore and restoration is needed to see boundaries, without the costly iterative optimization of earlier joint methods.

Relation to Prior Work

Several lines of work motivate the gap. SFT-GAN conditions super-resolution on a semantic probability map through a spatial feature transform, a per-pixel learned scale and shift applied to intermediate features. It showed that this beats concatenating the map at the input. Because its scale and shift depend only on class probabilities, two touching instances of one class receive identical modulation. Close the Loop (UBCN) couples deraining and semantic segmentation bidirectionally on Cityscapes and shows that the loop beats chaining a derainer before a segmenter. Its masks are categorical, so touching cars remain one region. Ren et al. allow motion discontinuities in video deblurring only at semantic boundaries, again leaving same-class contacts unprotected, and they report failure when blur defeats the initial segmentation. We take UBCN's loop and SFT-GAN's case against concatenation as motivation. What we change is the conditioning signal (instance contacts rather than classes) and where it acts (gating which pixels are mixed rather than modulating feature values).

The boundary-aware separation literature supplies our representation. Fields of The World labels each pixel as interior, boundary, or background for agricultural parcels. TernausNetV2 predicts a touching-border channel for buildings. Eschweiler et al. predict centroids and membranes in 3D confocal stacks and seed a watershed from them. All of these operate on output masks and degrade when blur dissolves the 1–2 pixel contact. Ceb classifies candidate boundary patches to fix watershed over-segmentation, but it depends on the initial probability map and takes about 4.8 seconds per image for signature extraction alone. That cost is a constraint: our boundary mechanism must be continuous and differentiable, with any watershed used only at evaluation.

The closest bridge is NASDM. It conditions a diffusion model, a generator that learns by gradually denoising random noise into samples, on a nuclear class mask plus an instance edge map, and it shows this prevents generated nuclei from clumping. NASDM, however, generates images from clean, given masks. It does not restore real degraded images, and it does not predict boundaries. We test whether its insight survives in the inverse problem, where boundaries must themselves be inferred from corrupted data. We also test whether concatenating an edge map, NASDM's mechanism, is enough there. CM-UNetv2 uses the phrase "boundary restoration," but it sharpens segmentation-mask edges rather than pixel intensities, so it is not a restoration baseline.

Research Program

The model has two parts. The first is a lightweight boundary head that predicts interior, contact, and background probabilities from the degraded image. The second is a restoration backbone whose local attention blocks are biased by that prediction.

Let $b_p \in [0,1]$ be the predicted contact probability at pixel $p$. For a query pixel $i$ and a neighbour $j$ in a local window, we define a crossing cost $B_{ij} = \max_{p \in \overline{ij}} b_p$. This is the largest contact probability on the straight segment between them, precomputed as a fixed set of pixel offsets per window position. The block output is

$$ y_i = \sum_{j \in \mathcal{N}(i)} \operatorname{softmax}_j\!\left(\frac{q_i^\top k_j}{\sqrt{d}} - \lambda B_{ij}\right) v_j , $$

where $q, k, v$ are the usual query, key, and value features of dimension $d$, $\mathcal{N}(i)$ is a 7×7 window, and $\lambda \ge 0$ is learned per layer. The term $\lambda B_{ij}$ acts like a soft wall. Pixels on opposite sides of a likely contact contribute little to each other, while pixels within one instance mix freely. Setting $\lambda = 0$ recovers an ordinary local-attention restorer, which gives a clean ablation. The max is subdifferentiable, so the boundary head trains end to end without any watershed.

The refinement pass works as follows. After a first restoration pass, the boundary head is re-run on the restored image. A second, weight-shared pass then uses the updated field. Training combines an L1 restoration loss with a weighted three-class cross-entropy on both boundary predictions, using ground-truth instance labels converted to interior, contact, and background. We will report results with one and two refinement passes to measure the latency and accuracy trade-off.

A known risk is that wrong boundaries create hard walls that split real objects or draw fake edges. This is the instance analogue of SFT-GAN hallucinating the wrong class texture. We will measure it directly by corrupting the boundary field at test time.

Baselines

The naive reference points come first. One is to run a fixed instance segmenter directly on the degraded image with no restoration. The other is to segment the clean ground-truth image, which gives the ceiling. For the clean ceiling on BBBC039, Ceb reports 95.2% F1, which bounds what any restore-then-segment pipeline can reach. For the degraded settings we construct, no reported result was found on Lacuna for degraded BBBC039 or degraded TissueNet, so we will produce these numbers ourselves.

The sequential baseline is the same restoration backbone with $\lambda = 0$, trained with L1 loss, followed by the same segmenters. These are a three-class U-Net with seeded watershed in the style of Eschweiler et al. and Cellpose, both trained on clean images and frozen.

The strongest existing coupled method is UBCN. On Cityscapes with synthetic rain, UBCN reports 77.09% mIoU and 43.22 dB PSNR. The sequential RCDNet derainer in the same paper reaches 76.13% mIoU. We will reimplement a UBCN-style loop on our backbone with categorical masks in place of the instance boundary field, so the only difference is the conditioning signal. On Cityscapes we will also run the published setting to check that our reimplementation lands near 77.09%. Beating UBCN means matching its mIoU within noise while improving instance-level AP for cars and persons and reducing the merge rate of touching same-class pairs.

The baselines that isolate our key ingredient use the same predicted boundary field injected in three cheaper ways. The first is NASDM-style input concatenation. The second is SFT modulation from the three-class map, which is SFT-GAN with instance-contact classes. The third is an L1 loss upweighted on ground-truth contact bands, with no boundary input at all. An oracle variant that receives ground-truth boundaries gives an upper bound for gating.

All variants share parameter count within 5%, training data, degradation pipeline, iterations, and a fixed hyperparameter search of equal size, and they are run with three seeds. Our method counts as winning only if it improves contact-band metrics and per-instance intensity error beyond seed variance over every boundary-injection baseline, without losing global PSNR by more than a small margin.

Why not just…?

Why not just restore with an off-the-shelf network and then segment? UBCN's own comparison shows chaining loses about a point of mIoU to coupling (76.13% vs 77.09% with RCDNet) even for semantic labels. We expect a larger loss at thin same-class contacts, which L1 restorers treat as blur to be smoothed. This is not assumed. The sequential baseline above measures it directly through merge rate and contact-gap contrast.

Why not just train the instance segmenter directly on degraded images and skip restoration? This is the most serious threat. For producing masks alone it may match or beat us, and we will include a degradation-augmented three-class segmenter as a baseline. If it wins on instance F1, we will say so. The proposal's claim then narrows to what masks cannot give: restored intensities that do not leak across contacts. This is measured as per-instance mean-intensity error computed inside ground-truth masks. A robust segmenter on a leaky restored image, or on the degraded image itself, still yields contaminated per-cell measurements.

Why not just concatenate a boundary map to the input, as NASDM does? NASDM shows concatenation works for generation from clean masks. SFT-GAN, however, reports that input concatenation of semantic maps is washed out by early layers, and Byun et al. found that naively merging channels lowered Cellpose F1 from 0.7607 to 0.7336. Neither result concerns predicted, noisy boundaries in restoration, so the concatenation ablation decides it. If concatenation matches gating, the mixing mechanism is unnecessary and we will report the simpler method.

Why not just upweight the restoration loss at contacts? This needs no boundary prediction at test time and could plausibly teach the network to keep gaps. It is included as a baseline. Our prediction is that it helps contact-gap contrast on clean training-like degradations but not under stronger or shifted degradation, where only an explicit, re-estimated boundary signal can adapt. The degradation-severity sweep tests exactly this.

Proposed Experiments

Data and degradations. We use BBBC039 nuclei and TissueNet nuclear and membrane images with instance labels. We apply controlled degradations: Gaussian and motion blur, Poisson–Gaussian noise, and 2× and 4× downsampling, at five severity levels. Training uses one range of severities, and the highest severities are held out to test shift. Cityscapes with synthetic rain, using its instance labels, connects us to UBCN's reported numbers.

Real-degradation check. The 3D confocal meristem stacks of Eschweiler et al. degrade naturally with depth but have no clean target. On them we report only downstream Jaccard index as a function of depth, using their seeded-watershed pipeline (reported 0.870 overall) on restored versus raw slices. We will also run a Fields of The World experiment with additional synthetic 2× downsampling of Sentinel-2 tiles, testing whether restoration recovers object recall on contiguous parcels. Small parcels there are already limited by mixed pixels.

Metrics. Restoration is scored by PSNR and SSIM computed both globally and within a 3-pixel band around ground-truth contacts. Contact-gap contrast is the intensity dip at the contact line relative to the adjacent interiors. Leakage is measured by per-instance mean-intensity error. Separation is scored by downstream instance F1 and AP from the frozen segmenters, plus two pair-level rates: the merge rate, meaning the fraction of ground-truth touching pairs fused into one object, and the split rate, meaning ground-truth instances divided by spurious walls.

Decisive checks. The first check is whether boundary gating beats concatenation, SFT modulation, and contact-weighted loss on contact-band metrics and leakage at matched capacity. If it does not, the core claim fails. The second is whether the gain grows with degradation severity and survives the held-out severities. The third is robustness to wrong boundaries: we perturb the field at test time by deleting 20% of contacts and adding spurious ones inside interiors. If split rate rises faster than merge rate falls, gating is trading one error for another, and a confidence-dependent $\lambda$ becomes necessary. The fourth is whether one refinement pass recovers most of the oracle-boundary gain at a fraction of the cost of a UBCN-style multi-iteration loop, reported as wall-clock time per image. The fifth is the head-to-head against the degradation-trained segmenter, which settles whether restoration's value here lies in masks, in measurements, or in both.