Most Fisheye Disocclusion Holes Were Already Seen: Resolution-Aware 4D Reconstruction for Surround-View Rigs

Cars and robots use fisheye cameras, lenses that see 180° or more by bending light heavily toward the image edge. Most 3D reconstruction methods expect ordinary undistorted images. "Rectifying" a fisheye image to fit them stretches the sparse edge pixels into blur and crops away the outer field of view. Recent work avoids this by building the fisheye lens model directly into Gaussian splatting, a method that represents a scene as millions of small colored 3D blobs and renders them by projecting each blob onto the image. These native-fisheye methods only work on scenes where nothing moves. Methods that handle moving objects, in turn, assume ordinary pinhole cameras. When a moving car is removed from a reconstruction, they leave holes and ghosts where the car used to be. The obvious fix is to bolt a fisheye projection onto a dynamic method and run a video inpainter over the holes. We think that fix misreads where the errors come from.

Our central claim is that in a surround-view rig, a patch of road hidden by a passing car in one camera at one moment has usually been seen by some camera at some other moment. Often it was seen near that camera's sharp optical center. Most "disocclusion" holes and most frame-to-frame flicker therefore come from a model that mixes sharp and badly compressed observations of the same surface without telling them apart. They do not come from missing information. We propose a fisheye-native 4D Gaussian reconstruction that keeps a per-blob record of the finest ground resolution at which each blob has ever been observed. That record drives three decisions: when to add blobs, what counts as a moving object, and which background regions are truly never-seen. Only the truly never-seen regions get filled by a generative model. The fill is done once, inside the static 3D layer, so every rendered frame shows the same content.

Core Claim

Each pixel of a fisheye image covers a different solid angle, and this solid angle grows sharply toward the image edge. Let $\omega_c(p)$ be the solid angle covered by pixel $p$ of camera $c$. It is computed from the calibrated lens polynomial $r(\theta)=a_1\theta+a_2\theta^2+a_3\theta^3+a_4\theta^4$ used by WoodScape. For a static Gaussian $g$ seen at depth $d_{g,c,t}$ by camera $c$ at time $t$, we keep a running "resolution ledger"

$$s_g=\min_{(c,t)\,:\,g \text{ visible}} \; d_{g,c,t}\,\sqrt{\omega_c(p_{g,c,t})},$$

which is the best ground sampling distance, in metres per pixel, of any unoccluded observation of $g$. Here $p_{g,c,t}$ is the pixel the Gaussian projects to, and "visible" means it is not blocked by a dynamic Gaussian at that time. The claim has three testable parts.

First, gradient-driven densification should be gated by $s_g$. Without the gate, a compressed peripheral view produces large blurry errors that the densifier tries to fix with extra blobs. Second, a large error between a blurry peripheral view and the sharp model is expected and should not be classed as a moving object. The transient test should compare residuals only against what the ledger says that view could resolve. Third, after removing dynamic objects, the set of background points with no valid observation at any time from any camera is small and explicitly computable. That set, and only that set, is where generative fill is needed.

Relation to Prior Work

Three papers supply the components we reuse. Fisheye-GS puts an equidistant fisheye projection and its gradients into Gaussian splatting. It shows that undistort-then-splat loses about 5 dB (21.69 vs 26.42 PSNR on the ScanNet++ Utility Room scene). ParkGaussian replaces the first-order projection with an Unscented Transform, which projects a few sample points per blob instead of a linear approximation, on four-camera parking-garage rigs. Both are explicitly static. We reuse the Unscented-Transform projection with the generic polynomial lens model so that real calibrations, not just ideal equidistant lenses, are handled.

On the dynamics side, Split4D gives blobs linear velocities and learned identities and separates moving objects without a 2D tracker. RobustSplat shows that delaying densification lets the static background settle before transient errors can spawn floating blobs. Both assume pinhole cameras.

Split4D also motivates the proposal directly: it reports ghosting, incomplete revealed background, and lingering shadows after objects are removed. RobustSplat motivates our densification gate. Its insight is that densification misreads large temporary errors as missing geometry, and fisheye periphery creates exactly such large errors without any motion at all.

The 2D perception analog of our ledger is FishBEV. It learns an uncertainty that down-weights high-distortion image edges when fusing surround cameras into a bird's-eye-view map. Our version differs in three ways. It is geometric rather than learned. It lives on 3D primitives rather than a planar grid. It is accumulated across time as well as across cameras, and the time accumulation is what turns it into a disocclusion tool.

Video inpainting work constrains our fill design. TSAM warps features between frames with optical flow and masks unreliable flow using cycle consistency. It reaches 37.43 PSNR on the Free-form Video Inpainting benchmark (FVI) with curve masks. It works in 2D on pinhole video. Its own reliance on flow is the weakness under fisheye distortion, and nothing ties its output to the 3D views a renderer will request later. The decoupling of "warp what is visible, hallucinate what is not" in Balakrishnan et al. is the template we follow. In our version, the "warp" stage is replaced by the 3D model with its ledger, and the "hallucinate" stage is restricted to a set computed from geometry rather than from a 2D mask.

Research Program

The system is a 4D Gaussian model with a static layer and a velocity-equipped dynamic layer, following Split4D's linear-motion parameterization. It is rendered through an Unscented-Transform fisheye projection using each camera's calibrated polynomial. Training follows RobustSplat's delayed-growth schedule, with three changes.

The first change is the densification gate. A static Gaussian may only split when the pixels driving its gradient have $d\sqrt{\omega}$ within a factor $\kappa$ of its current ledger value $s_g$. In other words, only views at least as sharp as the best one seen so far can ask for more detail.

The second change is resolution-aware transient scoring. Before computing the residual that flags moving content, the rendered static layer is blurred to the footprint implied by each pixel's $\omega_c(p)$. A compressed view is therefore compared with what it could have seen, not with the sharp model.

The third change is the fill step. After training, we compute the never-seen set $\mathcal{N}$: static-layer surface points, sampled on a bird's-eye ground plane, as in ParkGaussian's inverse perspective mapping, and on fitted vertical surfaces, whose ledger is empty or worse than a threshold. We render a single virtual pinhole or top-down view covering $\mathcal{N}$. We inpaint it once with an off-the-shelf image inpainter, using the observed surroundings as context. We then distill the result into static Gaussians with a loss restricted to $\mathcal{N}$. Because the fill lives in the static 3D layer, every later frame and viewpoint renders identical content there. Temporal consistency follows from the representation rather than from a temporal loss.

The analysis component comes first and is cheap. On real driving logs we will measure what fraction of each frame's dynamic-object footprint, projected onto the background, was observed unoccluded by any camera at any time within the sequence, and at what ground resolution. This number decides whether the paper's premise holds.

Baselines

The naive reference point rectifies each fisheye image to a cropped pinhole view and runs RobustSplat for the static background, or Split4D for the full decomposition. Any revealed holes are left as rendered. This is what a practitioner would do with existing code. Fisheye-GS's static result gives a reported reference for the cost of rectification (21.69 vs 26.42 PSNR on Utility Room). No reported result was found on Lacuna for rectified dynamic reconstruction on fisheye driving data.

The strongest competitor is the plug-in combination. It uses the same Unscented-Transform fisheye projection, Split4D-style velocities, and RobustSplat delayed growth, but no ledger. Disocclusion holes are filled by running TSAM on the rendered background video. It is the strongest because it contains every published ingredient that addresses each direction separately. If our gains come from simply having a fisheye projection plus dynamics, this baseline captures them.

On the static ParkRecon3D benchmark, the anchor for this projection family is ParkGaussian's reported 30.09 PSNR on Scene 1, against 28.70 for 3DGUT and 25.12 for OmniRe. Our static-only configuration must match ParkGaussian's number to show the ledger costs nothing when nothing moves. For the dynamic fisheye setting itself, no reported result was found on Lacuna for any method.

All methods will use identical calibrations, initial point clouds, iteration counts, and blob budgets. Each method's two most sensitive hyperparameters get the same grid-search budget on a held-out sequence. Beating the plug-in means four things on dynamic data. Revealed-region PSNR and LPIPS must improve with confidence intervals that exclude zero across sequences. The outer radial band must improve, not just the image center. Static-layer flicker must be lower. And there must be no regression on the full image.

Three ablations isolate the key ingredient. The first removes the ledger and keeps everything else. The second replaces the ledger with a simple per-pixel loss weight proportional to $1/\omega_c(p)$. This is the cheap "Jacobian reweighting" version, which uses the same geometry but no memory across time and cameras. The third keeps the ledger but fills holes frame-by-frame in 2D instead of once in 3D.

Why not just…?

Why not just rectify the images and run an existing dynamic method? Rectification blurs and crops exactly the outer image band where surround rigs see most of the road beside the car. Fisheye-GS measured a roughly 5 dB loss from undistortion even in static scenes. Our naive baseline will report the same comparison in the dynamic setting, broken down by radial band, so the loss is located rather than asserted.

Why not just plug a fisheye projection into RobustSplat or Split4D and stop there? This is the real threat, and it is our main baseline rather than a strawman. It could well recover most of the full-image PSNR gain. Our claim is narrower and testable. Without the ledger, compressed peripheral views will still trigger needless splits and false transient flags, and holes will be filled by a 2D inpainter where sharp observations from other cameras or times existed. The ledger-off ablation, scored on outer-band and revealed-region pixels separately, settles it. If the gap there is within noise, the proposal reduces to an engineering combination and we will report it as such.

Why not just weight the loss by each pixel's distortion? This is the cheapest form of resolution awareness and may explain much of the effect on visible regions. It cannot identify which background points have ever been seen sharply. So it cannot shrink the inpainting set or stop a blurry peripheral view from overwriting detail learned from a sharp one at another time. The second ablation tests exactly this. If it matches the full ledger on revealed regions, the cross-time memory is unnecessary and we will say so.

Why not just inpaint rendered frames with a video inpainter? TSAM's flow alignment is what makes it temporally stable, and flow is least reliable under fisheye distortion. More fundamentally, a 2D video inpainter has no way to make a new viewpoint agree with frames it already filled. The third ablation and the static-layer flicker metric measure this directly.

Proposed Experiments

The decisive pre-study runs on KITTI-360, which includes fisheye cameras, LiDAR, and separate static and dynamic object labels (KITTI-360), and on WoodScape's four 190° cameras with motion segmentation and LiDAR. We project every labeled dynamic object's footprint onto LiDAR-derived background surfaces and compute the fraction observed unoccluded elsewhere in the sequence. We record this both overall and at a ground resolution at least as good as the occluding view. If this fraction is low (say, under half), the central claim fails, and the project becomes a generative-fill paper instead.

Ground truth for revealed regions comes from a paired synthetic set. We use CARLA with the four-camera 190° fisheye configuration used for Synwoodscapes in FishBEV, and render each sequence twice, with and without traffic actors. Reconstructions are trained on the with-actor video. Their static layers are then scored against the actor-free renders from held-out poses using PSNR, SSIM, and LPIPS. Scores are reported separately for three pixel groups: revealed-and-seen-elsewhere, never-seen, and always-visible, and within each group by radial band.

Temporal consistency is measured by rendering the static layer from a fixed virtual camera at every training timestamp. Any pixel variation over time is error, because the static layer should not change. We also compute Video Fréchet Inception Distance (VFID), the video-level distribution metric used by TSAM, on rendered fly-throughs. On real WoodScape and KITTI-360 sequences, which have no actor-free ground truth, we hold out one camera per sequence and score only pixels labeled static. We also report the ratio of blob count to quality, to check the densification gate's predicted reduction in peripheral blob growth.

Finally, on static data we check for regressions. On ParkRecon3D Scene 1 the target is ParkGaussian's reported 30.09 PSNR. We also evaluate on FIORD, whose LiDAR ground truth allows a geometry check alongside the reported 3DGS figure of 25.31 PSNR for COLMAP initialization on Building_In.

The idea is confirmed if three things hold. The ledger-on model beats both the plug-in and the distortion-weighting ablation on revealed-and-seen-elsewhere pixels and in the outer radial band. Its never-seen set is substantially smaller than the 2D hole masks the plug-in hands to TSAM. And static-layer flicker drops by construction, without a temporal loss.