Seams at the Edge of the Hole: Closing the Depth-Error Gap in Disocclusion Inpainting

When software turns an ordinary video into stereo 3D, or renders any new viewpoint from one image plus a depth map, it slides each pixel sideways by an amount set by its depth. The background regions that were hidden behind foreground objects are now exposed and have no source pixels. These are disocclusion holes, and a learned inpainter fills them. Inpainters are trained and scored where the hole mask matches the geometry exactly. In real footage the depth comes from a monocular estimator and is wrong by a few pixels at object edges. The hole mask is then misplaced. Foreground colour gets smeared onto revealed background as a halo, and the inpainter fills the wrong pixels. We argue that much of the gap between clean-geometry and real-geometry performance lies in a thin band around the hole border. That band is a seam-selection problem: for each pixel, decide whether to keep the warped real pixel or the generated one.

The proposal is a training-free post-process. First, run any disocclusion inpainter on a slightly enlarged mask, so that it produces a candidate fill for the border band. Then solve a binary min-cut, a classic graph optimisation that finds the cheapest boundary, over that band. Each pixel chooses between the warped source and the generated proposal. The cost function prefers cuts where the two candidates agree, and it penalises keeping warped pixels where depth edges and image edges disagree. This is the energy-based seam optimisation used in image stitching and object compositing. Here it is pointed at the boundary between observed and hallucinated content, and it is given a depth-aware cost that 2D stitching energies lack.

Core Claim

The claim has three parts. First, the drop in disocclusion quality from ground-truth depth to estimated depth is concentrated in a narrow border band. Second, a single global fix of that band (a fixed mask dilation) cannot be right everywhere, because the depth error varies along the edge. Third, a depth-aware min-cut seam recovers a substantial share of the gap without retraining. The seam keeps real pixels wherever they are trustworthy and generates only where they are not.

This also explains a finding from novel view synthesis. Complexity-Stratified Evaluation Reveals Shortcut Regimes in Rotational Novel View Synthesis shows that copying the source view beats a conditional diffusion model on simple objects. The reported VGG perceptual distances are 0.157 for copying and 0.359 for the diffusion model. On complex objects copying collapses, at 0.520 against 0.288 for retrieval. Neither strategy decides locally where to copy and where to generate. A seam makes exactly that decision, pixel by pixel.

Relation to Prior Work

The method builds on the disocclusion literature, which motivates the problem and supplies the inpainters.

Geometric Reciprocity: Unlocking Self-Supervision for Stereoscopic Video Generation proves a reciprocity theorem: the hole mask for rendering a target view equals the mask of pixels lost when warping back to the source. This lets the authors train inpainters on real monocular video rather than synthetic stereo pairs, which they explicitly identify as a synthetic-to-real gap. Their masks are exact with respect to the depth used for warping, not with respect to the true scene. The authors themselves note sensitivity to depth errors, and the theorem assumes rectified, Lambertian geometry. We take their inpainter as our base model and attack the residual error they leave open.

A divide-and-conquer hole-filling method for handling disocclusion in single-view rendering handles edge stretching with a fixed rule: dilate the mask on the background side, ramp the depth, then median-filter. Our seam replaces that fixed rule with an optimised, spatially varying boundary.

Beyond the Visible: Disocclusion-Aware Editing via Proxy Dynamic Graphs and AMICO: Amodal Instance Composition blend revealed content with continuous latent or alpha compositing. AMICO reports overly smooth junctions as a result. We use a discrete cut that keeps every surviving pixel exactly as observed.

The seam machinery comes from compositing and stitching. Image editing by object-aware optimal boundary searching and mixed-domain composition routes a dynamic-programming boundary around salient structures using colour and gradient energies. Optimal Region Gradated Fusion Based Automatic Image Stitching for Industrial Systems confines blending to a narrow band around an optimal seam. It reports an average gradient of 28.06 against 7.36 for Laplacian blending on KC1600 roads, which is evidence that seams preserve high-frequency detail that blending destroys. Both methods use purely 2D energies and assume the inserted content is complete. We add a depth-reliability term and apply the cut where one side is generated.

The sim-to-real literature constrains the design. Understanding Domain Randomization for Sim-to-real Transfer shows that randomisation guarantees fail when deployment conditions fall outside the training distribution. That warns us that augmenting inpainter training with synthetic depth noise covers real estimator errors only as well as the noise model matches them. We test this directly rather than assume it.

The Seam Energy

Let $\hat D$ be the estimated depth, and let $M$ be the hole mask it induces. Let $B$ be a band of width $r$ around $M$'s border. The inpainter is run on $M$ dilated by $r$, giving a generated image $G$; $W$ is the forward-warped source image. Each pixel $p$ in $B$ gets a label $x_p$, either "warp" or "gen". Pixels inside the eroded hole are fixed to gen, and pixels outside $B$ are fixed to warp. We minimise

$$E(x)=\sum_{p\in B} U_p(x_p)+\lambda\sum_{(p,q)\in\mathcal N}[x_p\neq x_q]\,\big(\|W(p)-G(p)\|+\|W(q)-G(q)\|\big).$$

Here $\mathcal N$ is the set of 4-connected neighbour pairs and $\lambda$ weights seam smoothness against the per-pixel costs. The pairwise term is the standard stitching seam cost: cutting is cheap where the two candidates already agree.

The unary cost $U_p(\text{warp})$ is high where $p$ was warped from near a depth discontinuity of $\hat D$ that is not supported by an image edge in the source. It is also high where a left–right consistency check on $\hat D$ fails. $U_p(\text{gen})$ is a small constant. With binary labels and non-negative pairwise weights, min-cut solves this exactly. For a band a few pixels wide this costs milliseconds, which preserves the 0.05 s/frame speed that the reciprocity paper reports.

Research Program

The work has three stages.

The first stage measures where the gap lives. On Middlebury scenes with real second views and ground-truth disparity, we render each target view twice with the same inpainter: once with ground-truth disparity and once with disparity from a monocular estimator. We decompose the PSNR and LPIPS difference into contributions from the hole interior, the border band at several widths $r$, and the rest of the image. If the band does not hold a large share of the gap, the premise fails and the project stops here.

The second stage builds the seam on top of the reciprocity-trained inpainter. We tune $\lambda$, $r$ and the unary weights on a held-out split.

The third stage stratifies results by edge complexity, following the complexity-stratified evaluation in the novel-view-synthesis paper above. Simple straight depth edges may need no seam. Thin, curved or textured edges are where a spatially varying cut should matter.

Baselines

The do-nothing reference is the reciprocity-trained inpainter applied to the mask from estimated depth, with no border treatment. It anchors the gap measurement. On DAVIS-GRT, its paper reports PSNR 35.52 for the image model and 34.06 for the video model, against roughly 28–29 for StereoCrafter and StereoDiffusion. We will reproduce it first.

The strongest existing competitor is the same inpainter with the best fixed mask dilation. We tune the dilation radius on the same validation split, with a grid search budget equal to the one we spend on the seam's parameters. This is what practitioners actually do, and it uses the same generated pixels as our method. Beating it shows that spatially varying selection matters, not just "generate a wider band".

We also run the divide-and-conquer depth preprocessing as a hand-designed geometric fix. On Middlebury Flowerpots it reports a 2.55 dB PSNR gain over Xu's method; that paper gives no number for the reciprocity setting. Gradient-domain and feathered blending across the fixed border serve as the continuous-blending alternatives.

Two ablations isolate the key ingredient. The 2D-only seam keeps the pairwise term but sets the depth unary to zero, which tests whether the depth-aware cost matters beyond stitching-style energies. The unary-only variant thresholds $U_p$ without the pairwise term, which tests whether the cut's spatial coherence matters.

All methods share the inpainter weights, the depth estimator, the data, and the validation budget. Beating the tuned-dilation baseline means a statistically significant gain, over scenes, in band-restricted PSNR and LPIPS. It also requires no loss in full-image metrics, plus a lower halo score: the fraction of border pixels whose colour matches the foreground side.

Why not just…?

Why not just dilate the mask by a tuned radius? Dilation regenerates every border pixel by the same amount. Where depth was accurate it overwrites correct real pixels with hallucinated ones; this is the over-generation failure documented in the copy-versus-diffusion comparison. Where the depth error exceeds the radius it still leaves a halo. The stratified experiment settles this. If tuned dilation matches the seam on complex edges as well as simple ones, the seam adds nothing, and we will report that.

Why not just retrain the inpainter with random mask jitter and depth noise? This is the most serious threat. It is domain randomisation of the geometry, and it is cheap. The randomisation analysis predicts that it helps only to the extent that the synthetic noise covers real estimator errors, which are structured and concentrated at specific edges. We will train a jitter-augmented inpainter under matched compute and compare four systems: plain, augmented, plain plus seam, and augmented plus seam. If augmentation alone closes the band gap and the seam adds nothing on top, we will conclude that the problem is better solved in training.

Why not just blend the border with Poisson or feathering? These methods average or diffuse across the boundary rather than choosing which pixels to keep. The compositing paper documents colour bleeding and texture blurring from Poisson editing, and the stitching paper shows the loss of sharpness under blending (average gradient 7.36 versus 28.06). Near a misplaced depth edge, averaging mixes the halo into the background instead of removing it. The blending baselines and the halo score test this directly.

Why not just use a better depth estimator? Better depth shrinks the gap but does not remove it, and the seam composes with any estimator. We will repeat the main comparison with two estimators of different quality and report whether the seam's gain shrinks in proportion to the depth error. A shrinking gain is the expected signature if the seam is correcting depth-induced mask errors rather than something else.

Decisive Experiments

The primary test uses Middlebury real stereo with ground-truth disparity. The question is what fraction of the gap between ground-truth and estimated depth the seam recovers in band PSNR and LPIPS, relative to tuned dilation and to jitter-augmented retraining.

The secondary test is DAVIS-GRT. The seam must not reduce the reported 35.52 PSNR, and should raise border-band fidelity.

On Inria 3DMovie, which has no reported numbers on Lacuna, we will run a blind paired preference study focused on object boundaries. We will also measure seam-induced flicker by comparing frame-to-frame seam stability against the reciprocity paper's video model.

A clean negative result is also informative. That would be a gap that is not concentrated in the band, or one that augmentation fully closes. Either would show that depth-error robustness in disocclusion belongs in training, not in post-hoc boundary optimisation.