Splatting Through the Lens: Per-Channel, Depth- and Field-Dependent Blur for 3D Gaussian Splatting

Cheap, thin lenses such as flat metalenses (lenses made of a single patterned layer of nanoscale pillars) and Fresnel singlets blur each color channel differently. Red, green and blue focus at different distances, and the blur also changes with object depth and with how far the object sits from the image center. Today the only way to build a 3D scene model from such cameras is to deblur every photo in 2D first, which takes seconds per frame and gives answers that disagree between views, or to ignore the problem and let the 3D model absorb the blur. We propose to put the measured lens blur inside 3D Gaussian Splatting (3DGS), a scene representation built from millions of small colored 3D ellipsoids that are projected onto the image and blended. The key observation is simple. If the lens blur for one color channel is approximated as a small mixture of Gaussians, then blurring a projected Gaussian splat is exact and cheap: you add the blur kernel's covariance to the splat's 2D covariance. Each splat already knows its depth and its screen position, so a blur that varies with depth and field position costs nothing extra per primitive. In image space, by contrast, that kind of shift-variant blur requires expensive patchwise or depth-layered convolution.

We train with this blur model on, so the Gaussians themselves stay sharp, and at test time we drop it. Rendering then runs exactly as fast as plain 3DGS. The claim is narrow and testable. When the lens blur varies across color channels, depth and field position, a calibrated blur applied per splat recovers sharper, more view-consistent scenes than three alternatives: 2D pre-deblurring, learned blur absorption, and image-space convolution with a single blur kernel. It does so with no cost at render time and a modest training overhead. When the blur is the same everywhere, we expect no advantage over image-space convolution, and we will show that too.

Core Claim and Mechanism

Let a Gaussian project to a screen-space mean $\mu_{2D}$ and covariance $\Sigma_{2D}$, at depth $z$ and image position $u$. For sensor channel $\lambda \in \{R,G,B\}$, we calibrate the lens point spread function (PSF), meaning the image of a point of light, on a grid of depths and field positions. We fit it as a nonnegative mixture of $K$ Gaussians with weights $w_{\lambda,k}$, offsets $m_{\lambda,k}(z,u)$ and covariances $P_{\lambda,k}(z,u)$, and interpolate these over $(z,u)$. Convolving the splat with component $k$ gives another Gaussian:

$$\mu'_{\lambda,k} = \mu_{2D} + m_{\lambda,k}(z,u), \qquad \Sigma'_{\lambda,k} = \Sigma_{2D} + P_{\lambda,k}(z,u),$$

and its opacity is rescaled to conserve energy:

$$\alpha'_{\lambda,k} = \alpha \, w_{\lambda,k} \sqrt{\det \Sigma_{2D} / \det \Sigma'_{\lambda,k}}.$$

The offsets $m$ capture lateral color shift, where colors land at different image positions. The covariances $P$ capture the per-channel blur size and the axial focal shift, which depends on depth. During training we rasterize each channel separately with its own blurred footprints, because color channels no longer share one footprint. This means $3K$ passes instead of one, and we report that training cost explicitly. At test time we set $P=0$ and $m=0$, which gives a single standard pass.

This differs from its two closest relatives in what is known and what is learned. Mip-Splatting already adds a fixed Gaussian to $\Sigma_{2D}$, but its filter stands in for an idealized pixel box. It is the same for every channel and ignores the lens. That paper supplies the covariance-addition and opacity-rescaling technique we reuse. Deblurring 3D Gaussian Splatting uses a small network during training that enlarges Gaussian scales to absorb defocus, then discards it at test time. We adopt that discard-at-test design. However, its enlargement is learned, shared across red, green and blue, and unconstrained by optics. Our blur is measured, differs per channel, and is tied to depth and field position. That paper motivates the work by showing that 3DGS degrades badly on blurred input; we extend its premise to chromatic blur.

Two approximations need to be stated openly. First, blurring each splat before blending is not identical to blurring the final image at occlusion boundaries. Deblurring 3DGS makes the same approximation, and we measure the error near edges. Second, the PSFs of extended depth of focus lenses have ringed, axicon-like tails, as in Extended Depth of Focus Metalenses for Achromatic Computational Imaging and Inverse designed extended depth of focus meta-optics. A nonnegative Gaussian mixture cannot represent the rings. We will report the fraction of PSF energy the mixture misses and fold the residual into a small image-space correction term, so the approximation's cost is visible rather than hidden.

Research Program

The first component is optics calibration and simulation. For synthetic data we simulate three lenses with wave optics: a hyperboloid metalens, which has a large chromatic focal shift and narrow bandwidth; a shifted-axicon extended depth of focus metalens, following the designs in the extended depth of focus paper; and a refractive singlet. For each lens we compute per-channel PSFs over a grid of depths and field angles that includes off-axis positions. In that region color blur couples with coma and astigmatism, and the single-convolution assumption behind Wiener and total-variation deconvolution breaks down.

For real data we build a Fresnel or singlet camera, the lens class studied by Revisiting Cross-Channel Information Transfer. We calibrate it with a point source scanned over depth and field position, and capture multi-view scenes alongside a reference capture through a corrected lens.

The second component handles point cloud irregularity, which enters as an initialization constraint rather than a co-equal topic. Structure-from-motion (SfM), the step that recovers camera poses and a sparse 3D point cloud by matching features across photos, degrades on chromatically blurred images. The metalens burst paper reports that standard feature matching fails on such captures without special alignment. We initialize SfM from the sharpest channel after a fast per-channel Wiener filter, following the cross-channel paper's insight that one channel stays sharpest. We then fill holes with the k-nearest-neighbor compensation of Deblurring 3DGS. Because the resulting point clouds are sparse and uneven, every method is also run on deliberately thinned initial point clouds. This shows whether the calibrated blur makes optimization more robust to poor seeding.

Baselines

The naive reference is vanilla 3DGS trained directly on the aberrated photos. It shows how much blur and color fringing the representation silently absorbs. As an anchor on achromatic blur, Deblurring 3DGS reports that plain 3DGS reaches 21.66 dB PSNR on the real motion-blur set of the Deblur-NeRF benchmark, against 26.61 dB for its own method.

The decoupled pipeline deblurs each frame in 2D with Wiener deconvolution, and separately with cross-channel transfer, before running SfM and vanilla 3DGS. Cross-channel transfer reports 34.72 dB on a synthetic Fresnel 2D benchmark at about 7 seconds per 1.4-megapixel image. It is the strongest 2D chromatic restorer in the file.

The strongest existing 3D method is Deblurring 3DGS, run on the same initialization and with the same iteration budget. We also run a stronger variant that we construct ourselves, in which its blur network outputs separate scales and screen offsets for each color channel. This variant isolates "learned per-channel blur" from "calibrated per-channel blur."

The most dangerous baseline is an image-space forward model. It renders a sharp image, convolves each channel with the calibrated on-axis PSF via FFT, and compares the result to the photo. We also run a shift-variant version that uses patchwise PSFs and depth layers, with the number of patches and layers tuned to match our training time.

Ablations collapse our model step by step: a single shared PSF for all channels, then per-channel PSFs without depth dependence, then per-channel PSFs without field dependence. Every method gets the same data, the same camera poses, the same tuning sweep of three learning-rate settings and the same GPU-hour budget.

On our chromatic benchmark itself, no reported result was found on Lacuna for any method. Beating the field means at least 1 dB PSNR over the best baseline on the off-axis, depth-varying test views, a lower LPIPS (a learned perceptual distance), test-time frame rate within 5% of vanilla 3DGS, and training time at most 3 times that of vanilla 3DGS.

Why not just…?

Why not just deblur each frame in 2D and then run ordinary 3DGS? This approach is slow: about 2 seconds per frame for Wiener or total-variation deconvolution according to the extended depth of focus paper, and about 7 seconds for cross-channel transfer. More importantly, frames restored independently need not agree with one another. Deblur-NeRF gains about 5 dB over a NeRF trained on blurry views, 28.77 dB versus 23.78 dB, by modeling blur in 3D. That figure compares against no deblurring rather than 2D pre-deblurring, though, so the file does not settle this question. Our decoupled-pipeline baseline does. It also measures how often SfM fails without our sharpest-channel seeding.

Why not just let Deblurring 3DGS learn per-channel scales? This is a real threat on lenses with simple, nearly uniform blur. It is the reason we built the per-channel learned variant. Our expectation is that a learned scale factor attached to each Gaussian has no reason to reproduce the lateral color shift or the systematic growth of blur with field angle. Without calibration, it can trade blur against Gaussian size, leaving soft geometry behind. If the learned variant comes within 0.3 dB of our method on off-axis views, then calibration is not worth its setup cost, and we will report that.

Why not just convolve the rendered image with the measured PSF? For an on-axis, depth-invariant extended depth of focus PSF this is exact and cheaper than our method, and we expect it to tie or win there. The proposal's value rests entirely on the shift-variant regime. There, an image-space approach needs patchwise PSFs and depth layering, while splat-space blur is evaluated once per primitive. The time-matched shift-variant baseline is the decisive comparison.

Why not just buy an achromatic lens? Hardware-corrected broadband metalenses run into a trade-off between aperture, bandwidth and field of view, and stay near sub-millimeter apertures, as discussed by Wirth-Singh et al. and Xie et al.. The cheap large-aperture cameras that robotics wants are exactly the ones that leave strong, shift-variant color blur for computation to fix.

Proposed Experiments and Decisive Checks

We evaluate on synthetic versions of the Blender and Mip-NeRF 360 scenes rendered through the three simulated lenses, where ground-truth sharp images allow full-reference PSNR, SSIM and LPIPS. We also report cross-view color consistency. On the real singlet captures, we compare against the reference-lens captures after registration. To check that the method does no harm on achromatic blur, it runs on the Deblur-NeRF defocus sets with a channel-shared PSF fitted to Gaussian defocus.

Four checks decide the outcome. First, on the on-axis extended depth of focus lens, image-space convolution should tie our method; a large win for us there would point to a bug, not a strength. Second, on off-axis views of the hyperboloid lens and the singlet, our method should beat the time-matched shift-variant image-space baseline by at least 1 dB. Third, removing field dependence should cost most of that gain, which would confirm that field-dependent blur is the ingredient doing the work. Fourth, as the initial point cloud is thinned, our method's quality should degrade more slowly than vanilla and learned-blur 3DGS. That would show that a physically correct blur model reduces the floaters and oversized Gaussians that P2ENet and Point As Gaussian produce on sparse input.

We will also report training passes, peak memory, and test-time frame rate on a desktop GPU and on a Jetson Orin. The Jetson comparison follows the edge setting of Photo-SLAM, which reports 100 FPS on that device.