Layered Video Stitching: Let the Warp Tear at Depth Edges, Then Fill the Tears

Stitching video from two moving cameras into one wide view forces a choice. The warp, meaning the function that bends one camera's image onto the other's, can track every frame's exact alignment, which makes the panorama jitter. It can be smoothed over time, which makes it stable, but then nearby objects show up twice or smeared because their alignment changes faster than the smoothed warp allows. The current best online system, StabStitch, reports this tradeoff directly: buying temporal stability costs some alignment accuracy, and large depth differences remain a failure case (Eliminating Warping Shakes for Unsupervised Online Video Stitching). This proposal argues that the tradeoff is mostly self-inflicted. It comes from asking a single continuous warp to register a near foreground and a far background at the same time. Parallax, the effect where near things shift more than far things when the camera moves, makes those two registrations disagree.

The idea is to split each frame into a few depth layers and give each layer its own warp and its own temporal smoothing. Each layer is roughly planar, so its correct warp really does change smoothly over time, and smoothing no longer fights alignment. The price is that the combined warp is now allowed to tear at depth boundaries. That opens holes where background revealed behind a foreground object was not seen by the reference view; these are called disocclusions. In two-camera video, most of those holes can be filled with real pixels, either from the other camera at the same instant or from earlier frames tracked along the stable background trajectory. Only a small remainder needs a generative model. Put plainly, the bet is that visible, fillable holes are cheaper than distortion or ghosting, and that the multi-camera, multi-frame setting makes holes unusually easy to fill.

Core Claim

Inside a depth layer, trajectory smoothing and parallax alignment do not conflict. Gap 2 in the research file names the "smoothness versus abrupt depth transitions" tension as unresolved. We claim that tension disappears once the warp is decomposed by depth and the resulting disocclusions are treated as a retrieval problem first and a generation problem second. The measurable prediction is specific. On high-parallax video, a layered, tearing warp plus retrieval-first filling should match StabStitch-style stability and beat it on alignment and ghosting. The advantage should grow with how much the scene's depth varies, and it should vanish on near-planar scenes.

Relation to Prior Work

Three lines of work supply the method. StabStitch builds "stitching trajectories" by chaining each camera's motion over time with the warp between cameras, and smooths them with a small 3D convolutional network over a sliding window (Eliminating Warping Shakes for Unsupervised Online Video Stitching). StabStitch++ warps both streams halfway onto a virtual middle plane using a global homography plus a thin-plate spline, a smooth bendable grid, so that distortion is split evenly between them (StabStitch++). We reuse both machines, one copy per layer. M2SVid fills disocclusion masks in stereo video with a single-step refiner adapted from Stable Video Diffusion; it applies full space-time attention only on hole pixels so that it can copy matching content from other frames (M2SVid). We reuse that design as the last-resort filler.

Two lines motivate the layering but stop short of it. Segmentation-guided multi-homography stitching assigns separate homographies to separate regions, but only for still images and with no temporal model (Parallax-tolerant Image Stitching via Segmentation-guided Multi-homography Warping). Plane-based content-preserving warps show that per-plane motion models reduce depth-dependent wobble in single-camera stabilization (Plane-Based Content Preserving Warps for Video Stabilization). Neither combines per-layer smoothing with cross-camera alignment, and neither has to deal with the holes that tearing creates.

Two lines constrain the design. Flow-guided video inpainting breaks down when optical flow is wrong at occlusion boundaries (Deep Flow-Guided Video Inpainting), and flow-based latent propagation only works between adjacent frames (Flow-Guided Diffusion for Video Inpainting). For that reason we propagate fill content along the background layer's smoothed parametric trajectory rather than along dense optical flow, since that trajectory is well-defined exactly where flow fails. Generative panoramic stitching anchors diffusion outpainting to global canvas coordinates and handles 1–2 m baselines (Generative Panoramic Image Stitching). It needs per-scene fine-tuning and many sampling steps, and it is frame-independent, so we treat generation as the remainder step rather than the main mechanism. Learned elastic warps already show tearing and holes as a failure (Learning Residual Elastic Warps for Image Stitching under Dirichlet Boundary Condition). Here the tear is placed deliberately at depth edges and then repaired.

Research Program

Layer estimation. For each frame we estimate depth, using monocular depth refined by stereo matching inside the camera overlap. We cluster depth into $K\in\{1,2,3\}$ layers and keep the masks consistent over time by propagating them with the previous frame's per-layer warps. Masks are soft mattes a few pixels wide, so seams between layers can be blended rather than cut.

Per-layer warps and smoothing. Each layer $k$ gets a StabStitch++-style midplane warp with trajectory $T_k(t)$, smoothed by its own network instance. The per-layer objective is

$$ \mathcal{L}_k=\sum_t \big\|M_k(t)\odot\big(I_A(T^A_k(t))-I_B(T^B_k(t))\big)\big\|_1+\lambda_s\sum_t\|T_k(t+1)-2T_k(t)+T_k(t-1)\|^2+\lambda_d\,D(T_k(t)). $$

The symbols mean the following. $M_k(t)$ is layer $k$'s mask at frame $t$. $I_A$ and $I_B$ are the two camera streams sampled through their halves of the midplane warp. The second term penalizes trajectory acceleration, which is what produces warping shake. $D$ is StabStitch's shape-preservation penalty, which punishes departure from local affinity.

The design hypothesis is that the smoothness weight $\lambda_s$ can be set high for every layer without hurting alignment, because each layer's true warp is near-planar. A single-layer warp cannot do the same, since it has to fit depth steps. Composing the layers back to front yields a stitched frame plus an explicit hole mask.

Retrieval-first filling. Hole pixels are filled in a fixed priority order. The first source is the other camera at the same time, warped through the background layer's transform. The second source is the most recent past frames, at most $N=7$ to match StabStitch++'s window and stay online. These are reached by composing background-layer trajectories, and a photometric consistency check is applied in the same spirit as the forward-backward check in Deep Flow-Guided Video Inpainting. Whatever is still missing goes to an M2SVid-style single-step refiner, fine-tuned for stitching. Its attention over space and time is restricted to the residual mask, and it is conditioned on the retrieved context. Along the way we log what fraction of hole pixels each stage fills. That fraction is itself a finding: if retrieval covers most of the holes, generative hallucination stays a minor contributor.

Baselines

The naive reference is UDIS++ run independently on every frame (Parallax-Tolerant Unsupervised Deep Image Stitching). It is the strongest single-image unsupervised stitcher in the file, with PSNR 30.19 and SSIM 0.933 reported on UDIS-D easy cases, and its per-frame use is exactly what produces warping shake. We will use its released weights, plus test-time warp adaptation with the same per-frame step budget. Beating it means lower instability with no loss in overlap PSNR.

The strongest existing method is StabStitch++ (StabStitch++). It is the only online system that jointly handles the alignment-distortion tradeoff, parallax, and warping shake. The file lists no PSNR, SSIM, or stability numbers for it on StabStitch-D ("outperforms UDIS++", with numbers not given), so we rerun it ourselves. For its predecessor StabStitch, the file reports a stability score of 48.74 against 81.57 for per-frame stitching (Eliminating Warping Shakes for Unsupervised Online Video Stitching), and we will check that our reproduction lands in that range before trusting the comparison. Both systems train on the StabStitch-D training split with the same window length, the same training iterations, and the same hyperparameter search budget. Beating StabStitch++ means stability within its noise band together with clearly lower ghosting and higher overlap PSNR on the high-parallax subset.

The decoupled pipeline is StabStitch++ followed by flow-guided video diffusion inpainting (Flow-Guided Diffusion for Video Inpainting). The inpainter is applied to a mask of ghosted pixels, detected as large photometric disagreement in the overlap. This baseline isolates whether where the warp tears matters or whether a strong inpainter can clean up any warp afterward.

The 3D reprojection baseline estimates per-pixel depth and camera poses, reprojects both streams to a virtual middle camera, and fills holes with the same M2SVid-style refiner. It is the "skip layers, use full geometry" competitor.

Ablations. Setting $K=1$ removes layering while keeping the filler. Sharing a single smoothing network and trajectory across layers isolates per-layer smoothing. Removing the retrieval stages so that the generator fills every hole isolates retrieval-first filling. Sweeping $\lambda_s$ for $K=1$ against $K=3$ maps the stability-alignment curve directly. The central prediction is that the $K=3$ curve dominates the $K=1$ curve.

Why not just…?

Why not just run StabStitch++ and then inpaint the ghosted regions? This is the closest threat. A continuous warp spreads parallax error as soft double edges across a wide band, and a ghost detector must then delete correct pixels along with wrong ones, which enlarges the hole the inpainter has to invent. Flow-guided inpainters also degrade exactly at the depth boundaries where these ghosts sit (Deep Flow-Guided Video Inpainting). The decoupled baseline above tests this head-on. If it matches us on the held-out-pixel benchmark, the layering claim fails and we will report that.

Why not just use one global homography and let a video diffusion model fill every mismatch? The file's evidence counts against letting generation carry structure. RealFill's LoFTR matching score was 0.030 against 0.181 for a canvas-anchored model (Generative Panoramic Image Stitching). Unconstrained video generation reached 0.02 motion accuracy against 0.72 for geometry-grounded generation (Beyond the Visible). Our "no retrieval" ablation, run with a global homography, tests the stitching version of this question directly.

Why not just do full depth-based 3D reprojection? This is a real threat. If depth were accurate, layers would only be a coarse approximation of it. Our counter-hypothesis is that per-pixel depth errors at object edges produce tearing and stretching in the reprojected image, while a few near-planar layers average those errors out. The 3D reprojection baseline settles the question. We will report the comparison stratified by depth-estimation error. If reprojection wins whenever depth is good, the honest conclusion is that layers are a robustness device for low-texture and low-light footage, not a general improvement.

Proposed Experiments

Real-world stability and alignment. We evaluate on StabStitch-D with its Regular, Low-Texture, Low-Light, and Fast-Moving categories, using the StabStitch stability metric, PSNR and SSIM on the overlap, and the shape-distortion penalty. We add a high-parallax subset, chosen by the spread of estimated depth inside the overlap, because the hypothesis predicts gains there and none on near-planar clips.

Held-out ground truth for disocclusions. StabStitch-D has no ground truth for hole pixels, so we build a benchmark from Stereo4D stereo video (Stereo4D). We take a partially overlapping crop of the right-eye view and stitch it into the full left-eye frame. Disoccluded pixels, visible in the left eye but not in the right crop, then have exact ground truth: the left-eye pixels themselves. To make the setting a true stitching problem, the left frame's own pixels inside the hole region are withheld from the method. We report PSNR and LPIPS restricted to the hole mask, temporal warping error inside the mask, and the fraction of hole pixels filled at each stage. Static checks run on UDIS-D against the reported UDIS++ figure of 30.19 PSNR, using the per-frame version of our method; this confirms that layering does not hurt still-image alignment.

Decisive checks. The proposal succeeds only if all three of the following hold. On the high-parallax subset, the $K=3$ stability-versus-alignment curve must strictly dominate the $K=1$ curve across the $\lambda_s$ sweep. On Stereo4D, masked hole-region error must beat both the decoupled pipeline and 3D reprojection. Retrieval must fill most hole pixels, so that the generator handles a minority. If the first check fails, the core claim is wrong regardless of image quality. Runtime per frame will be reported against StabStitch++'s 35.3 ms per frame as a practical constraint, not as the main claim.