Most systems that build a 3D model of one object from photos need a cut-out first: a 2D mask saying which pixels belong to the object. When that mask is wrong, the error goes straight into the 3D model. Small leaks become floating "ghost" geometry, and missed pixels erase hair, wires, and leaves. The underlying problem is that a single ordinary photo cannot say how far away anything is. A pixel at an object's edge also records a blend of object and background light, and no 2D segmenter can undo that blend reliably. This proposal asks whether a camera that spends some of its pixels on seeing the scene from a few slightly different directions can make the mask unnecessary. Such a camera is a light-field camera: a microlens array or sub-lens array splits one sensor into many small viewpoints. With it, the separation of object from background would come from how rays behave across those viewpoints, all within a single exposure.
The central bet is specific. At a fixed sensor pixel budget, the best recipe for isolating and reconstructing an object with thin parts is neither the full-resolution single view nor the dense 9×9 plenoptic grid. It is a small interior number of views, and that optimum can be predicted from the width of the thinnest structure. We would build a reconstruction method that decides which 3D elements belong to the object by checking whether rays from different sub-apertures agree on them. We would then map the pixel-versus-view Pareto curve for object isolation, a curve the literature argues about but has never measured.
Core Claim
Sparse Light Field Sampling Improves Casual 3D and 4D Reconstruction showed that "parallax trumps resolution" for whole-scene reconstruction. On NeRF Synthetic, a single multiplexed exposure reaches 26.39 dB PSNR against 16.25 dB for a single monocular exposure. That work reconstructs the scene as one block and never isolates an object. The segmentation literature points the other way on thin structures. Mask-dependent reconstruction degrades on hair and fine silhouettes (Sebastian Winberg et al.), and dividing a sensor into $N \times N$ views shrinks every structure by a factor of $N$ in each sub-view (1 Light Field Vision and Coded Imaging).
Our claim resolves this tension with a measurable prediction. Consider a thin structure that is $w$ pixels wide at full sensor resolution. With $N \times N$ views it is $w/N$ sub-view pixels wide. For a fixed main-lens aperture, the total parallax across the aperture scales down by the same factor, so the ratio of parallax to structure width does not depend on $N$. What does change with $N$ is whether the structure is sampled at all. We therefore hypothesize that object-boundary accuracy peaks near
$$N^* \approx \max(2, w),$$
where the lower bound of 2 reflects that separating a blended pixel requires at least two directions. Reconstruction that pools sub-pixel information across views, as light-field super-resolution does (NTIRE 2024 Challenge on Light Field Image Super-Resolution), should push $N^*$ upward. Measuring by how much is part of the contribution. The method claim is that deciding object membership by cross-view ray agreement removes ghost geometry that 2D masks, per-view masks, and depth-based cropping all leave behind at thin boundaries.
Prior Work and Where This Sits
Three lines of work each supply one piece of the idea, and none combines the pieces.
The first supplies the capture model. The sparse light-field paper above supervises 3D Gaussian Splatting with a forward model of overlapping sub-lens images plus total-variation regularization. Gaussian Splatting represents a scene as many small colored 3D blobs rendered by projection. We reuse that forward model unchanged, and we add object/background separation, which it does not attempt.
The second supplies the evidence that ray space separates boundaries better than pixels. 4D Light Field Superpixel and Segmentation segments 4D ray bundles with an 80-neighbor spatial-angular graph and beats 2D superpixels on boundary recall. However, it works on dense Lytro light fields, requires precomputed disparity, and produces no 3D model. View-consistent 4D Light Field Depth Estimation enforces view-consistent depth along epipolar-plane lines, the 2D slices of a light field in which a scene point traces a line whose slope encodes depth, but it does no object segmentation. We move their ray-consistency reasoning inside a differentiable 3D reconstruction and test it at sparse angular sampling, where their dense-grid assumptions no longer hold.
The third supplies the mask-free motivation and the voting idea. ShapeR drops 2D masks in favor of SLAM points projected as "point masks" and reports a 2.7× Chamfer improvement. It needs camera motion, so it cannot work on one exposure. RayFusion removes ghost detections by intersecting ray occupancies across cooperating vehicles. We apply that intersection logic to sub-apertures inside one camera body. Among the human-reconstruction methods that motivate the problem, GaussianAvatar and DoubleField both report that mask errors turn into ghost geometry.
Coded-aperture depth estimation constrains the design rather than supplying it. Single-Image Depth from Defocus with Coded Aperture and Diffusion Posterior Sampling gets single-shot depth (0.025 m MAE on its benchmarks) but yields a 2.5D map, one depth value per pixel, with no way to model the background behind a thin foreground strand. That is exactly the case we target.
Research Program
We would build a two-layer Gaussian reconstruction from one multiplexed exposure: one set of Gaussians for the object and one for everything else. Both are rendered through the multiplexed forward model and summed through standard alpha compositing, so each sub-view pixel can be explained as a blend of object and background.
Membership is not given by a mask. Each Gaussian $i$ gets a support score
$$s_i = \frac{1}{V}\sum_{v=1}^{V} \mathbb{1}\left[\,|z_v(\pi_v(\mu_i)) - z_i| < \tau\,\right].$$
Here $V$ is the number of sub-aperture views, $\mu_i$ is the Gaussian's center with depth $z_i$, $\pi_v$ projects into view $v$, $z_v$ is the depth rendered in that view, and $\tau$ is a tolerance. Gaussians that only one or two views "need" are the ghosts a monocular fit creates, and they receive low support. Those Gaussians are pushed toward the background layer or pruned.
A single user click, or a coarse 3D box, seeds which high-support cluster is the target. This matches the weak grounding ShapeR uses, but without motion. The support score is made soft, using a sigmoid in place of the indicator, so that membership is learned jointly with geometry.
The analysis program sweeps the same simulated sensor budget across 1×1, 2×2, 3×3, 5×5, and 9×9 splits. It uses scenes with controlled thin-structure width $w$ and controlled foreground-background depth gap, and it tests whether the measured optimum tracks $\max(2,w)$.
Baselines
The naive reference point is the full-resolution single exposure. It is segmented with an off-the-shelf 2D foundation segmenter (SAM) and fit with masked monocular Gaussian Splatting. It represents the "just use better masks" position at the maximum spatial resolution the sensor allows. On whole-scene NeRF Synthetic, the unmasked version of this pipeline reaches 16.25 dB PSNR (Sparse Light Field Sampling). We found no reported masked, object-only result on Lacuna, so we will measure it.
The strongest existing competitor is the sparse light-field pipeline itself, with its reported 26.39 dB single-exposure NeRF Synthetic PSNR and 31.71 dB on real dynamic multi-view captures. We would run it at each pixel-budget split and isolate the object in two ways. The first applies SAM masks to every sub-view before fitting. The second reconstructs the whole scene and then crops Gaussians with the same seed click expanded to a depth-thresholded 3D region. This is the competitor most likely to match us, because it already has parallax and only lacks explicit membership.
A two-stage ray-space baseline runs 4D Light Field Superpixel and Segmentation on the sub-views and feeds its masks to the same reconstruction. This tests whether our joint formulation beats "segment in 4D, then reconstruct." A depth-only baseline back-projects DFD-DPS depth at the matching single-shot budget; its paper reports 0.025 m depth MAE.
The key ablation keeps our two-layer model but replaces the support score with a learned per-Gaussian membership that has no cross-view term. This isolates whether ray agreement, rather than the two-layer structure, does the work.
All methods share the same simulated or real exposure, the same Gaussian optimizer, iteration count, and densification schedule, the same seed click, and the same hyperparameter search budget on a held-out validation scene. Beating the strongest competitor means lower thin-structure boundary error and fewer ghost Gaussians at equal or better object PSNR, across a majority of depth-gap settings.
Why not just…?
Why not just run SAM on each sub-view and feed the masks to the light-field reconstruction? This is a real threat when objects are large and well separated. At thin boundaries, however, every sub-view pixel is still a blend of object and background. 4D Light Field Superpixel and Segmentation documents 2D boundary instability under defocus, and each sub-view is $N$ times lower in resolution than the image SAM was trained on. Independent per-view masks also disagree with each other, and GaussianAvatar reports that such disagreement becomes ghost geometry. This baseline is run directly. If it matches us on the thin-structure subset, the method claim fails and only the Pareto-curve contribution survives.
Why not just use a monocular depth model plus SAM on the full-resolution image? This option keeps every pixel, but it guesses depth instead of measuring it. The sparse light-field paper reports a 10 dB gap on NeRF Synthetic and roughly 50% lower relative depth error with even a ~5° baseline. DFD-DPS reports that learned monocular networks break under optical shift. A single depth value per pixel also cannot represent the background visible behind a hair strand. This is the 1×1 point of our sweep, so the curve itself answers the question.
Why not just reconstruct the whole scene with parallax and crop by depth? This is the most serious alternative. If the object sits well in front of the background, a 3D crop may be all that is needed. Cropping fails where object and background Gaussians interleave in depth: near contact points, for thin structures close to a wall, and for ghost Gaussians that happen to fall inside the crop volume. Our depth-gap sweep is designed to show where cropping and support-scoring diverge. We expect them to tie at large gaps and to separate as the gap shrinks. If they tie everywhere, we will report that the separation problem is solved by parallax alone, which is itself a useful negative result.
Why not just move the camera? Motion gives larger baselines, but it confuses object motion with depth in dynamic scenes; the sparse light-field paper reports monocular streams at 24.51 dB against 31.71 dB for synchronized views. Motion also cannot serve single-shot capture. ShapeR is the motion-based mask-free reference, but it cannot run on one exposure, so we cite its setting rather than compete in it.
Proposed Experiments and Decisive Checks
The main testbed is a cluttered NeRF Synthetic. We would render the ficus, ship, and lego objects in front of textured backdrops at controlled depth gaps, then simulate multiplexed capture at each pixel-budget split using the forward model of the sparse light-field paper. Exact object geometry and mattes give ground truth for three measurements. The first is object-only PSNR. The second is Chamfer distance to the ground-truth object. The third is boundary F-score inside a thin-structure trimap, a band marking pixels whose ownership is uncertain. We also measure a ghost rate: the fraction of object-layer Gaussians lying outside the ground-truth object volume.
The HCI 4D Light Field benchmark provides boundary recall and achievable segmentation accuracy against 4D Light Field Superpixel and Segmentation. No numeric value for it was found on Lacuna, so we rerun it. For real data, the Lytro Illum and iPhone scenes of the multi-view real-world dataset used in the sparse light-field paper supply held-out-view object PSNR, with object regions annotated by hand on held-out views only.
Three checks decide the proposal. First, the measured boundary-accuracy optimum over $N$ should shift with $w$ roughly as $\max(2,w)$ predicts. If it is flat, or always at 9×9, the Pareto hypothesis is wrong. Second, support-scoring should beat both per-view SAM masks and depth cropping on ghost rate and thin-structure boundary F-score at small depth gaps. If it does not, cross-view membership adds nothing beyond parallax. Third, the ablation without the cross-view term should lose most of that advantage; if it does not, the two-layer model rather than ray agreement is responsible.
References
- Sparse Light Field Sampling Improves Casual 3D and 4D Reconstruction
- GroomCap: High-Fidelity Prior-Free Hair Capture
- 1 Light Field Vision and Coded Imaging
- NTIRE 2024 Challenge on Light Field Image Super-Resolution: Methods and Results
- 4D Light Field Superpixel and Segmentation
- View-consistent 4D Light Field Depth Estimation
- ShapeR: Robust Conditional 3D Shape Generation from Casual Captures
- RayFusion: Ray Fusion Enhanced Collaborative Visual Perception
- GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians
- DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Reconstruction and Rendering
- Single-Image Depth From Defocus With Coded Aperture and Diffusion Posterior Sampling