When a 3D scene is reconstructed from only three or four photographs, the model can match every training image while inventing the wrong shape. It produces semi-transparent "floaters" in front of the cameras, and blank surfaces collapse into flat cards facing each camera. The failures are worst exactly where images carry no information: textureless walls, ceilings, glass. Sound behaves differently. A loudspeaker's echo off a blank wall arrives at a microphone array with a measurable delay and direction, and with standard image-source geometry, delay plus direction places the reflecting surface in 3D. We propose to turn those echoes into a sampling distribution: a probability, along each camera ray, of where a surface lies. This distribution decides where the reconstruction places and grows its primitives. The width of each acoustic hypothesis is set by how precisely the particular microphone array can resolve direction, so a small four-microphone device yields broad, cautious proposals and a large array yields sharp ones.
The reconstruction is 3D Gaussian Splatting, which represents a scene as millions of small colored ellipsoids that are rasterized at hundreds of frames per second. The opportunity is threefold. First, sparse-view reconstruction gets a physical geometry cue that does not come from a pretrained image network. Second, the cue is strongest precisely where image-based depth priors fail. Third, fewer spurious primitives mean a smaller, faster final model. Audio is used only at training time, so rendering latency is unchanged or improved.
Core Claim
Multi-channel early-reflection measurements, converted into per-ray surface proposals whose widths are calibrated to the array's geometry, reduce floaters and planar collapse in sparse-view 3D Gaussian Splatting on textureless regions. They do so more reliably than monocular-depth-guided sampling, and they produce fewer Gaussians and lower memory at equal or better image quality. Two corollaries are falsifiable. First, the benefit should shrink smoothly as array aperture and microphone count shrink, tracking the predicted angular resolution. Second, a single-channel impulse response, which lacks direction, should give most of its benefit only through a loss term and little through sampling.
Relation to Prior Work
The closest work falls into three groups.
The first group motivates the proposal by showing that audio can fix sparse-view geometry, but not how we intend to do it. NeRAF jointly trains a NeRF and an acoustic field through a shared voxel grid, so acoustic loss gradients regularize visual geometry. It reports up to 15% gains in few-shot PSNR and LPIPS. However, it uses single-point source and listener pairs, relies on slow volumetric ray marching, and uses audio only as a loss. We use the directional content of a multi-microphone array to decide where primitives go, which is a different lever. AV-GS densifies Gaussians on textureless planes because those planes matter acoustically. It runs in the opposite direction from us: vision serves audio synthesis, and it requires dense views to initialize.
The second group supplies the sampling machinery and sets the main competitor. Multimodal-Prior-Guided Importance Sampling for Hierarchical Gaussian Splatting gates sparse-view densification using photometric residuals, ResNet18 edge features, and depth and curvature from DPT, the pretrained monocular depth network. It reports +0.3 dB over NexusGS on 3-view benchmarks. Our proposal replaces the monocular depth term with an acoustic one, and the comparison between the two is the central experiment. TD-NeRF samples a truncated normal around monocular depth along each ray. DDNeRF shows that continuous per-interval Gaussians match mip-NeRF with a third of the samples (20.84 PSNR at 32 samples vs. 20.80 at 96). We reuse this per-ray truncated-mixture form.
The third group constrains the design through documented failures of geometric scaffolds. GPF samples densely in a band around a PatchmatchMVS point scaffold and cuts render time from 86 s to 1 s per frame. Its authors note that it fails where stereo fails on textureless walls and wide baselines, which is the same regime we target.
From the audio side, DeFTAN-AA, VarArray and PhaseCoder show that models can work across arbitrary microphone layouts. PhaseCoder in particular feeds explicit microphone coordinates into a transformer. None of these models connects to a 3D visual representation. We treat array-agnostic learning as an optional upgrade, not the core, for reasons given below. Acoustic Volume Rendering motivates working in the frequency domain, where delays become phase shifts and fractional-delay errors disappear. We adopt this for the delay estimation step.
Mechanism
For each recorded source position, we take the first tens of milliseconds of the multi-channel impulse response, where discrete wall reflections dominate. In that window we localize reflections with a steered-response-power search, which scans candidate directions and delays for the combination where the channels line up in phase, carried out in the frequency domain. Each detected reflection, combined with the known source position, defines a candidate reflecting surface patch via the image-source construction. We accumulate these patches into a sparse 3D reflector field. Each patch is a Gaussian with a mean position and a covariance. The covariance follows from the array, not from a learned network. The angular spread of a detection is approximately the diffraction limit of the aperture, and the range spread follows from signal bandwidth:
$$\sigma_\theta \approx \frac{\lambda}{D\sqrt{\mathrm{SNR}}}, \qquad \sigma_r \approx \frac{c}{B\sqrt{\mathrm{SNR}}},$$
Here $\lambda$ is the wavelength at the analysis band's center, $D$ the array aperture (largest inter-microphone distance), $c$ the speed of sound, $B$ the analysis bandwidth, and SNR the reflection's signal-to-noise ratio. These are the standard aperture and bandwidth resolution limits, not a fitted model.
Each camera ray $\mathbf{r}(t)$ then receives a proposal density over depth $t$:
$$p(t) \propto (1-w)\,p_{\text{vis}}(t) + w\sum_k \pi_k\,\mathcal{N}\!\left(t;\mu_k,s_k^2\right).$$
In this mixture, $p_{\text{vis}}$ is the current photometric depth distribution from the Gaussian model. Each acoustic patch $k$ intersecting the ray contributes depth mean $\mu_k$, width $s_k$ (the patch covariance projected onto the ray), and weight $\pi_k$. The mixing weight $w$ is computed per pixel. It is high where image texture is weak and where the patch's projected width is small, and near zero otherwise.
The density is used in two places. At initialization, Gaussians are seeded by sampling it inside the camera frustums. During densification, a proposed split or clone is accepted only if the new primitive lands where $p(t)$ exceeds a threshold. Under-supported primitives become candidates for pruning after a short protection period, the same device as Xiong et al.'s grace period. Rendering is ordinary Gaussian rasterization, so no audio runs at test time.
Research Program
We will build the estimator, the proposal, and the densification gate on top of a standard 3DGS codebase. Development uses simulation first. SoundSpaces 2.0 renders impulse responses on Replica and Matterport meshes with arbitrary virtual array layouts, which provides ground-truth geometry and lets us sweep array aperture, microphone count, and reflection SNR independently.
Real evaluation uses the Real Acoustic Fields dataset. It pairs a 36-microphone array (the Earful Tower), a robotic speaker stand, and a multi-camera rig (the Eyeful Tower) in furnished and unfurnished rooms. We subsample its imagery to 3 and 6 views. Subsampling its 36 microphones to subsets of 1, 2, 4, 8, and 16, chosen to span small and large apertures, gives a controlled study of array geometry sensitivity on real hardware. RAF also reports that simulator pretraining plus about 100 real samples matches full real training. That result suggests the optional learned components can be trained mostly in simulation.
The analysis has two parts. First, we check whether the array-derived widths are calibrated, meaning whether true wall depths fall inside $\pm 2 s_k$ at the predicted rate. Second, we check whether reconstruction gains track $\sigma_\theta$ across array subsets. If the gains do not follow the resolution prediction, the mechanism is not what we claim.
Baselines
The naive reference is vanilla 3DGS with standard residual-driven densification on the same sparse views. It shows the raw floater and memory problem.
The strongest existing vision-only competitor is the multimodal-prior-guided hierarchical 3DGS of Xiong et al.. It is the most direct rival: it targets exactly sparse-view densification with geometric priors. The file reports only its +0.3 dB margin over NexusGS on 3-view benchmarks and no absolute PSNR on any dataset. We found no reported result on Lacuna for any method on sparse-view RAF imagery, so all RAF numbers will be produced by us. We will run it with its released three-phase schedule. As a second vision baseline we will run depth-aware sparse 3DGS (Few-Shot Novel View Synthesis Using Depth Aware 3D Gaussian Splatting). For an anchor on DTU, the file reports RegNeRF at 18.89 PSNR with 3 views (RegNeRF). DTU has no audio, so it only checks that our method with $w=0$ does not regress relative to that NeRF-family reference.
The strongest audio competitor is a NeRAF-style acoustic regularizer (NeRAF) ported to 3DGS. In this baseline the impulse-response loss backpropagates into a voxelized Gaussian density with no sampling guidance. It isolates the question of using audio as a loss versus as a proposal.
Fairness constraints are the same: same views, same training iterations, a matched hyperparameter search of 20 trials per method on a held-out scene, and the same GPU. All audio-using methods get the same impulse responses.
Two ablations isolate the key ingredient. The first replaces array-calibrated widths with a single fixed width. The second uses the 1-microphone subset, which has delay but no direction.
Beating the strongest baseline means three things on RAF 3-view, averaged over scenes. Masked PSNR and depth error on textureless regions must improve with confidence intervals excluding zero. Whole-image PSNR must not drop. Gaussian count and VRAM must be equal or lower.
Why not just…?
Why not just use a better monocular depth prior, as Xiong et al. and TD-NeRF do? This is the real threat. On textured surfaces a modern depth network may be as good as echoes, and it needs no extra hardware. The evidence against it is local rather than global. TD-NeRF needed truncation, an inter-frame point constraint, and a return to uniform sampling to survive DPT noise, and GPF documents scaffold failure on textureless walls. We therefore report every metric separately on textureless and specular masks versus textured regions. We also run a fused variant that uses both priors. If the monocular prior matches us on the textureless masks, the acoustic proposal is not worth the hardware, and we will say so.
Why not just add a single-channel impulse-response loss, as NeRAF does? A single channel gives echo delay but not direction. Delay alone constrains surfaces to an ellipsoid around source and receiver, which cannot place a wall along a specific camera ray. The 1-microphone ablation and the NeRAF-style baseline test this directly. Our prediction is that they help mainly through global room-volume constraints and fall short of the array proposal on per-ray depth.
Why not just train an array-agnostic network like PhaseCoder to predict geometry end to end? This might eventually work, but PhaseCoder and DeFTAN-AA are trained for localization and enhancement, not surface mapping. A learned predictor would also lose the explicit, auditable link between array aperture and proposal width that makes the method safe to apply to an unknown device. We keep classical steered-response localization as the primary estimator. We train a coordinate-conditioned set encoder in the style of PhaseCoder only as a replacement for the reflection detector. It is adopted only if it improves calibration on simulated arrays held out from training.
Proposed Experiments and Decisive Checks
The first decisive check is calibration in SoundSpaces 2.0. Across randomized arrays, the fraction of ground-truth wall depths inside the predicted $\pm 2s_k$ band should be near 95%. Gross miscalibration would invalidate the sampling design before any rendering is done.
The second is the RAF 3-view and 6-view comparison against all baselines. It reports PSNR, SSIM, and LPIPS on held-out views, rendered-depth error against RAF's reconstructed geometry where available, results stratified by texture mask, final Gaussian count, VRAM, rendering frame rate, and training time.
The third is the array sweep: performance as a function of microphone subset. The claim survives only if gains rise monotonically with aperture and approximately follow $\sigma_\theta$.
The fourth is an adversarial check on furnished rooms, where clutter produces spurious reflections. It tests whether the per-pixel weight $w$ correctly suppresses acoustic proposals. If it does not, floaters should appear behind furniture, and an increase there relative to the vision-only baseline would count against the method.
References
- NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis
- Multimodal-Prior-Guided Importance Sampling for Hierarchical Gaussian Splatting in Sparse-View Novel View Synthesis
- TD-NeRF: Novel Truncated Depth Prior for Joint Camera Pose and Neural Radiance Field Optimization
- DDNeRF: Depth Distribution Neural Radiance Fields
- Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation
- DeFTAN-AA: Array Geometry Agnostic Multichannel Speech Enhancement
- VarArray: Array-Geometry-Agnostic Continuous Speech Separation
- PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs
- Acoustic Volume Rendering for Neural Impulse Response Fields
- SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
- Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark
- Few-Shot Novel View Synthesis Using Depth Aware 3D Gaussian Splatting
- RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs