A light field camera, such as a plenoptic camera or a camera array, records a scene from a dense grid of slightly shifted viewpoints in one shot. Today, turning that capture into a renderable 3D scene means one of two things. The first is per-scene optimization: fitting 3D Gaussian Splatting (3DGS), a representation made of many small colored, semi-transparent blobs that can be rendered very fast, to all the views by gradient descent, which takes minutes per scene. The second is running a feed-forward reconstructor built for ordinary photos. Those reconstructors predict one or more Gaussians per pixel of every input view. With a 9×9 grid that means 81 copies of nearly the same surface, and memory grows with the number of views. They also find depth by matching features between views, and that matching was designed for wide camera spacings, not the sub-pixel shifts between neighboring light field views.
We propose a feed-forward network that reads a whole light field and outputs one compact set of 3D Gaussians in under a second. Its central idea is that the cheap, factorized convolutions light field researchers already use can do two jobs at once. Spatial convolutions run within each view. Angular convolutions run across views at a fixed pixel, and if the light field is first sheared by a candidate disparity, they become a depth test. The angular step then collapses the 81 views into a single Gaussian per spatial location, carrying both a depth distribution and view-dependent color. The number of primitives therefore depends on image resolution and not on view count. Duplicate primitives are never created in the first place, so no clustering step is needed to remove them afterwards.
Core Claim
Spatial-angular separable convolution replaces a dense 4D kernel over the two image axes $(H,W)$ and the two viewpoint axes $(U,V)$, which costs $K^4$ weights per channel, with a 2D spatial kernel plus a 2D angular kernel, which costs $2K^2$. So far it has been used only for tasks that map light field images to light field images. Fast Light Field Reconstruction with Deep Coarse-to-Fine Modeling of Spatial-Angular Clues alternates spatial and angular convolutions to synthesize missing views. End-to-End View Synthesis for Light Field Imaging with Pseudo 4DCNN factorizes processing into row and column networks over epipolar-plane stacks. Both output sub-aperture images, and neither produces 3D primitives. Their known weakness is the "approximation gap": oblique structure in ray space, meaning large disparities and steep epipolar slopes, is poorly captured by axis-aligned factorized kernels.
Our claim is that shearing the 4D volume before the angular convolution closes this gap and turns the angular branch into a depth estimator. For a candidate disparity $d$, the shear is the resampling $L_d(x,y,u,v)=L(x+du,\,y+dv,\,u,\,v)$. It aligns every scene point at disparity $d$ along the angular axes, so a small angular kernel at fixed $(x,y)$ sees a constant signal exactly when the hypothesis is right.
Running the angular convolution over a small set of hypotheses $\{d_k\}$ and taking a softmax gives a per-pixel distribution over depth buckets. That is the same quantity pixelSplat predicts with epipolar cross-attention. Here it is obtained with structured convolutions on the regular viewpoint grid. The angular features at the selected hypothesis also encode how color changes with viewpoint, so they directly supply each Gaussian's view-dependent appearance. The output is one Gaussian per spatial location, or a few per location in a coarse-to-fine cascade. The new element is this mapping from the factorization of the input to the structure of the output.
Research Program
The encoder is a stack of alternating spatial and angular convolutions over a $U\times V$ grid of sub-aperture images, following the interleaving design of Yeung et al., with two changes. First, the angular layers act on sheared copies of the feature volume. Second, the shear hypotheses are refined coarse-to-fine, borrowing the cascade from OccCasNet. OccCasNet searches a wide disparity range coarsely and then a narrow range at 1/8-pixel steps, and it masks occluded rays using warped side views. We reuse both ideas inside the network. Instead of returning a 2.5D depth map for the center view, we pass the result to a Gaussian head.
The Gaussian head operates on a reference image grid. For each location it outputs a depth bucket distribution, with opacity tied to the bucket probability as in pixelSplat, along with scale, rotation, and opacity. Color is a function of viewing direction, computed from the angular features. We will compare a third-order spherical-harmonic color with a small learned angular basis. The comparison matters because LiTo reports that third-order spherical harmonics blur sharp highlights.
To recover surfaces hidden from the center view but visible from side views, the occlusion mask lets the network emit a second, lower-density layer of Gaussians where side views disagree with the center depth. The total primitive count is therefore bounded by a small multiple of $H\times W$, never by $U\times V\times H\times W$.
Training uses a rendering loss on held-out viewpoints. To give that loss real 3D leverage, we train on synthetic light fields whose held-out targets include viewpoints outside the capture grid. We render grids at several baselines from 3D assets and from synthetic scenes. We fine-tune on real Lytro and gantry captures, because Light Field Super-Resolution: A Benchmark reports about a 1.25 dB drop when synthetic-trained light field networks meet real data. As an optional second stage, we test whether two ReSplat-style feedback steps help. In such a step, rendering errors on the input views are fed back without gradients, as in ReSplat. Our network computes the update with separable convolutions rather than kNN attention. This stage is secondary and is reported separately.
Baselines
The naive reference point has two parts. The first is the center view lifted to Gaussians at a constant depth. The second, and more meaningful, is depth-then-splat: predict center-view disparity with OccCasNet, unproject each pixel to one isotropic Gaussian colored by that pixel, and render. This tells us how much of the problem is solved by good light field depth alone. OccCasNet is reported to rank first in MSE and Q25 on the HCI 4D benchmark at 1.13 s per image (OccCasNet). We use the published model and the same image resolutions. To beat this baseline, our method must win clearly on held-out views near occlusion boundaries and on specular regions, not just on average.
The strongest feed-forward competitor is DepthSplat. It fuses a plane-sweep cost volume with Depth Anything V2 priors and reconstructs 12 views at 512×960 in 0.6 s. We give it the sub-aperture images with their exact grid poses, retrain it on our synthetic light field training set under the same GPU-hour budget, and run it at 3×3, 5×5, and 9×9 grids. We found no reported result on Lacuna for DepthSplat, or any feed-forward 3DGS model, on light field data, so all its numbers will come from our runs. Beating it means at least matching its held-out-view PSNR and LPIPS while using an order of magnitude fewer Gaussians and less peak memory at 9×9. It also means a clear depth-accuracy win (BadPix on HCI) at sub-pixel disparities.
For pure in-grid view synthesis, the strongest image-to-image method is Pseudo 4DCNN, reported at 43.28 dB average PSNR on real Lytro Illum scenes at 0.28 s per view. We run it on the same angular-upsampling protocol. We expect it to be hard to beat inside the grid because it never commits to 3D. Matching it within roughly 1 dB there, while also supporting extrapolated views and real-time rasterization that it cannot provide, would count as success.
The quality ceiling is per-scene 3DGS optimized on all input views. We report it with a time-versus-PSNR curve so the speed-quality trade-off is explicit, not hidden.
Three ablations isolate the key ingredient. The first replaces our angular collapse with per-view pixel-aligned Gaussians from the same encoder, which isolates primitive fusion. The second removes the shear, giving plain separable convolutions, which isolates the fix for the approximation gap. The third replaces the separable stack with a reduced-width full 4D convolution at matched memory, testing whether factorization costs accuracy.
Why not just…?
Why not just run DepthSplat, or any multi-view 3DGS model, on the sub-aperture images and prune afterwards? This is the most serious threat. The light field is, after all, a posed multi-view set. The case against it is structural. Pixel-aligned prediction grows linearly with view count, which is why AnySplat needs differentiable voxelization to prune 30% to 70% of primitives and ReSplat subsamples to 1/16 of the points. Plane-sweep matching is also reported to fail on reflective and textureless surfaces (DepthSplat). Neither argument proves it will lose at narrow baselines. The 3×3 to 9×9 sweep, with memory, Gaussian count, and BadPix measured side by side, settles it. If DepthSplat with voxel pruning matches us on all three, the contribution reduces to efficiency, and we will say so.
Why not just predict depth with a light field depth network and splat the center view? That is our naive baseline, and for diffuse interiors it may come close. It cannot represent surfaces occluded in the center view, and it has no view-dependent color. The proposal therefore stands or falls on the occlusion-band and specular-region breakdowns. OccCasNet itself notes failures on untextured and specular areas.
Why not just optimize 3DGS per scene, given that 81 views is plenty of supervision? For offline quality it may win. The question here is whether one forward pass gets close in a fraction of the time. ReSplat shows a learned recurrent update can surpass 4,000-iteration 3DGS while running 100× faster on DL3DV. Our time-to-quality curve shows where the feed-forward model sits against optimization stopped at matched wall-clock time.
Why not use a global transformer like GS-LRM over all views? GS-LRM handles 2 to 4 views at up to 512×904, and its attention cost is quadratic in tokens. With 81 views that is infeasible without heavy downsampling. We include a GS-LRM-style baseline on a 3×3 grid at matched parameters. If it wins at 3×3, the data still show whether separable operators are needed only for scaling, or for accuracy as well.
Proposed Experiments
Evaluation covers the HCI 4D benchmark (synthetic, with ground-truth disparity), the NTIRE-LFSR light fields (real Lytro plus synthetic), EPFL and Lytro Illum real captures, and Stanford Gantry for wider baselines. Synthetic scenes also come with extrapolated viewpoints rendered outside the capture grid. Metrics are PSNR, SSIM, and LPIPS on held-out in-grid views and on extrapolated views. Depth is scored with MSE and BadPix on HCI. Efficiency is scored by Gaussian count, peak GPU memory, reconstruction time, and render speed. Quality results are broken down by occlusion bands and specular masks.
Three checks decide the claim. First, at 9×9, our Gaussian count must stay roughly constant as the grid grows from 3×3, while DepthSplat's grows about ninefold, at equal or better held-out PSNR. Second, the no-shear ablation must lose substantially on large-disparity HCI scenes while the sheared model does not. This directly tests whether shearing closes the approximation gap. Third, on extrapolated views, our explicit Gaussians must beat Pseudo 4DCNN, which can only interpolate inside the grid. They must also beat depth-then-splat in occlusion bands.
If the second check fails, the separable operators contribute only efficiency. If the first fails against pruned DepthSplat, the native-fusion argument does not hold. Either outcome would be reported plainly.
References
- Fast Light Field Reconstruction with Deep Coarse-to-Fine Modeling of Spatial-Angular Clues
- End-to-End View Synthesis for Light Field Imaging with Pseudo 4DCNN
- PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction
- OccCasNet: Occlusion-Aware Cascade Cost Volume for Light Field Depth Estimation
- LiTo: Surface Light Field Tokenization
- Light Field Super-Resolution: A Benchmark
- ReSplat: Learning Recurrent Gaussian Splats
- DepthSplat: Connecting Gaussian Splatting and Depth
- AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting