When a robot or survey rig scans a scene, two physical limits hit at once. Depth sensors see only the surfaces facing them, so the back and inside of objects are missing. Cameras face a trade-off between spatial and spectral resolution. A camera that splits light into many narrow wavelength bands, called a hyperspectral camera, collects so few photons per band that its pixels must be large and its images coarse. A colour (RGB) camera gets sharp images but only three broad bands. Recent methods fill in missing 3D geometry using a diffusion model, which is a generator trained to turn random noise into realistic samples by removing noise step by step. These methods work only in RGB or with synthetic normal maps. Methods that reconstruct hyperspectral 3D scenes assume every surface was seen. Nothing in between produces a complete 3D scene with physically meaningful spectra on the surfaces that no sensor observed.
The proposal rests on one observation. An RGB diffusion prior cannot invent a narrow absorption band it has never seen, but it can often tell which of a few known materials a hidden surface is made of. So we model the spectrum of every 3D primitive as a non-negative mixture of a small set of material spectra, called endmembers, extracted from the surfaces that were observed. Unseen regions are then completed by a pretrained 2D diffusion prior acting on geometry and on mixture weights, not on raw spectra. Each loss term is applied only in the region of space where it carries information. The result is a completed hyperspectral scene, and the method is designed so that we can measure whether this constraint actually beats simply painting hallucinated RGB with a learned colour-to-spectrum map.
Core Claim
The claim is that, in partially observed scenes, restricting hidden-surface spectra to the set of mixtures of observed endmembers turns spectral completion from an ill-posed regression into a low-dimensional choice that an RGB diffusion prior can make. Pairing this with region-gated losses avoids the conflicts between geometry and appearance gradients that otherwise produce floating artifacts and spectral drift.
Concretely, each 3D Gaussian $g$ (a small coloured, semi-transparent ellipsoid used in Gaussian splatting) carries mixture weights $a_g$ on the probability simplex. Its spectral radiance is a weighted sum of $K$ endmember spectra $e_k$. Its colour as seen by any camera is that spectrum integrated against the camera's spectral response $S_c$:
$$s_g(\lambda)=\sum_{k=1}^{K} a_{g,k}\, e_k(\lambda), \qquad c_g=\int S_c(\lambda)\, s_g(\lambda)\, d\lambda .$$
Because $c_g$ is a fixed linear function of $a_g$, a diffusion prior that scores rendered RGB images sends useful gradients straight to the material weights. Here $K$ is small, about 3 to 8, following the abundance-space argument below. The spectra themselves are never hallucinated, only selected and mixed.
Relation to Prior Work
Two lines of work supply the machinery we reuse.
For geometry completion, ComPC completes a point cloud at test time. It initialises Gaussians from the partial scan, freezes the observed ones, grows new ones, and optimises them with score distillation from the Zero 1-to-3 diffusion model on normal-coloured renders. Score distillation means using a pretrained image diffusion model's denoising signal as a loss on rendered views. We reuse this almost unchanged for geometry. Our difference is that ComPC carries no radiometry at all, while we attach a constrained spectral model to every new Gaussian.
For geometry-appearance conflicts, EnerGS splits space into occupied, free and unknown regions using partial LiDAR. It pulls Gaussians toward occupied space with a robust Welsch loss, repels them from free space with a barrier potential, and stops photometric gradients from moving Gaussian positions. We borrow the partition and the barrier. EnerGS has no generative prior in unknown space, and it is RGB-only.
On the spectral side, HyperGS splats latent spectral vectors from a pretrained autoencoder and trains with Charbonnier plus cosine-similarity losses. It supplies our observed-region spectral loss and our strongest full-coverage baseline. Its free latent spectrum, however, has nothing to anchor it where no view exists, which is exactly what our simplex parameterisation replaces. SpectralNeRF likewise assumes dense calibrated views.
The low-dimensional parameterisation is motivated by Deep Diffusion Models and Unsupervised Hyperspectral Unmixing. It runs diffusion over 3 to 5 dimensional abundance maps instead of 200-band cubes and shows that this sidesteps high-dimensional instability. It does so in 2D satellite imagery, whereas we use abundances as the unknown in a 3D completion problem.
The fusion data term follows Hyperspectral Image Super-resolution with Deep Priors and Degradation Model Inversion. It inverts an explicit blur-and-downsample model linking a low-resolution hyperspectral image and a high-resolution RGB image, and we apply that same forward model per view to rendered spectra.
One paper constrains the design. DMGASR reports that training a spectral autoencoder and a latent diffusion model separately limits spectral-angle gains. That argues against a frozen-autoencoder-plus-separate-prior pipeline. It is also why our endmembers enter the rendering equation directly, with no decoder in the loop.
Research Program
Inputs. The system takes three kinds of data: partial LiDAR or depth, a set of high-resolution RGB views, and co-registered low-resolution hyperspectral views. This reproduces the spatial-spectral trade-off sensor by sensor. Large parts of the scene are visible in neither camera.
Material extraction. Blind unmixing of the observed hyperspectral pixels gives the endmembers $e_k$. These are then refined jointly during optimisation.
Losses, applied by region.
- Observed views. Rendered spectra are blurred and downsampled to match the hyperspectral frames using Charbonnier plus cosine losses. The same spectra are projected through $S_c$ to match the RGB frames.
- Occupied space. A Welsch pull toward the LiDAR points.
- Free space. The barrier potential.
- Unknown space. The only signals are score distillation on normal-coloured renders, which acts on geometry, and on RGB renders computed from the mixture weights, which acts on materials. Observed Gaussians are frozen as in ComPC.
The gating is the multi-component loss design. No term receives a global weight in a region where it has no measurements, so the number of weights that must be tuned against one another shrinks.
Pruning question. EnerGS blocks photometric gradients from moving Gaussian positions. HyperGS instead relies on joint gradients to place and prune spectrally important Gaussians. We test both settings, because which one wins under occlusion is an open question raised by the two papers.
Known failure mode. A material that appears only on hidden surfaces cannot be represented as a mixture of observed endmembers. We build a held-out-material condition specifically to measure this. We also add an optional residual endmember, penalised to stay small, to see whether this failure can be flagged when the RGB prior and the simplex fit disagree.
Baselines
Naive reference. HyperGS trained on the partial views with no completion. It shows how much of each hidden region stays empty or floats. On full-coverage Bayspec it reports 27.11 dB PSNR against HS-NeRF's 19.82 dB. No reported result was found on Lacuna for HyperGS under occlusion.
Strongest geometry competitor. ComPC run on the same partial input with the same Zero 1-to-3 prior. It reports a Chamfer Distance of 1.35 on Redwood, against 2.16 for the next-best supervised model. We give it the same Gaussian budget and the same number of distillation steps. Since ComPC has no spectra, we pair it with the spectral fill described next.
Strongest overall competitor. A two-stage pipeline: ComPC geometry and hallucinated RGB, followed by a per-scene RGB-to-spectrum regressor. The regressor is a small multilayer perceptron, or a continuous spectral decoder in the style of SSIF, trained on the scene's own observed RGB and hyperspectral pairs and applied to hidden surfaces. It gets the same observed data and the same tuning budget, a fixed grid search over the same number of configurations as our method. No reported result was found on Lacuna for any method in this occluded hyperspectral setting.
Geometry-loss reference. EnerGS losses with HyperGS appearance and no diffusion. It isolates what the generative prior adds. For calibration, EnerGS reports a KITTI Leak Score of 9.2, against 12.5 for vanilla 3D Gaussian splatting.
Ablations.
- Replace the simplex parameterisation with a free HyperGS-style latent spectrum in unknown space, keeping everything else fixed. This isolates the key ingredient.
- Remove region gating and apply all losses everywhere with tuned global weights.
What counts as beating the baselines. We must lower the spectral angle on hidden-surface pixels of held-out views relative to the two-stage pipeline, beyond seed-to-seed variation. We must also match ComPC's Chamfer Distance within its seed variance and not raise the leak score.
Why not just…?
Why not just complete geometry and colour with ComPC, then map RGB to spectra? This is the real threat. If a scene's materials have distinct RGB appearances, a regressor trained on that scene can recover hidden spectra nearly as well as our method. Our method should win in two cases. The first is metamerism, where materials share an RGB colour but have different spectra. A per-pixel regressor averages such materials into a non-physical spectrum, whereas the simplex combined with spatial coherence from joint optimisation should pick one. The second is when wrong hallucinated geometry would produce wrong colour that a two-stage method cannot revise. We will build test scenes with controlled metameric pairs. If the two-stage pipeline matches us there too, we will report that the simplex constraint is unnecessary and recommend the simpler pipeline.
Why not just copy spectra from the nearest observed surface? Nearest-neighbour propagation is fast, and it is right whenever a hidden back face matches the front face. It fails at material boundaries that lie inside the hidden region, for example a lid of a different material. We include it as a baseline and report hidden-region spectral angle stratified by whether the ground-truth material changes across the occlusion boundary.
Why not train a 3D spectral diffusion model directly? No corpus of complete hyperspectral 3D shapes exists in the material we draw on. Supervised completion networks trained on synthetic shapes also generalise poorly to real scans, as ComPC's Redwood comparison shows. Diffusing over raw hundred-band data is itself unstable and slow: DMGASR reports 23.01 s per 256×256 image for band-wise diffusion, against 6.17 s in latent space. A test-time 2D prior over a few mixture weights avoids both problems.
Proposed Experiments
Main benchmark. We use the ScanNetv200-derived synthetic hyperspectral scenes from HyperGS, where ground-truth spectra and geometry exist for every surface. We simulate incompleteness by removing a hemisphere of camera poses and cropping LiDAR to a limited vertical field of view. We simulate the sensor trade-off by downsampling the hyperspectral views spatially, as is standard in CAVE-style fusion studies, and keeping RGB views at full resolution.
Real data. On Bayspec and SOP we hold out views covering a chosen region. Ground truth then exists only in those held-out views, so geometry is scored against the full-view HyperGS reconstruction as a pseudo-reference.
Metrics. We report PSNR and spectral angle separately for pixels of observed and hidden surfaces, Chamfer Distance to ground-truth geometry, and the EnerGS leak score.
Decisive checks.
- Primary. Hidden-region spectral angle against the two-stage pipeline on the metameric-pair subset.
- Mechanism. The free-latent ablation, which shows whether the simplex, rather than the extra machinery, causes any gain.
- Gating. The gated versus ungated comparison, which tests whether region gating reduces floaters without hurting spectral pruning.
- Limits. The held-out-material condition, which measures how badly the method fails when its central assumption breaks and whether the residual endmember detects the failure.
Runtime is reported against ComPC's roughly 15 minutes per object. We do not claim real-time operation.
References
- ComPC: Completing a 3D Point Cloud with 2D Diffusion Priors
- EnerGS: Energy-Based Gaussian Splatting with Partial Geometric Priors
- HyperGS: Hyperspectral 3D Gaussian Splatting
- SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field
- Deep Diffusion Models and Unsupervised Hyperspectral Unmixing for Realistic Abundance Map Synthesis
- Hyperspectral Image Super-Resolution with Deep Priors and Degradation Model Inversion
- Enhancing Hyperspectral Images via Diffusion Model and Group-Autoencoder Super-resolution Network
- SSIF: Learning Continuous Image Representation for Spatial-Spectral Super-Resolution