TL;DR
VolSplat introduces a novel feed-forward 3D Gaussian Splatting paradigm that shifts from predicting Gaussians per-pixel to predicting them from a unified 3D voxel grid. This voxel-aligned approach eliminates reliance on error-prone 2D feature matching, enables adaptive control over Gaussian density based on scene complexity, and achieves state-of-the-art novel view synthesis with superior geometric consistency.
What Problem Does This Paper Address?
Feed-forward 3D Gaussian Splatting (3DGS) has become a popular approach for fast, single-pass 3D reconstruction and novel view synthesis without requiring costly per-scene optimization. However, existing methods predominantly rely on a pixel-aligned paradigm: they predict one 3D Gaussian primitive for every single 2D pixel across the input views.
The authors identify three major bottlenecks inherent to this pixel-aligned design:
- Rigid Density Distribution: The number of 3D Gaussians is strictly tied to the 2D image resolution. This forces a uniform distribution of primitives, causing severe over-densification (redundancy) on flat, textureless surfaces and under-densification in geometrically complex regions.
- Alignment Errors: Fusing multi-view information relies heavily on 2D feature matching. When source views contain occlusions or low-texture areas, this matching fails, leading to alignment errors and floating artifacts ("floaters").
- View Bias: The resulting 3D models remain heavily dependent on the specific viewpoints of the input images rather than forming a truly cohesive, view-independent 3D structure.
Methods
To decouple 3D reconstruction from the 2D pixel grid, VolSplat introduces a voxel-aligned framework. The pipeline consists of four main stages:
- 3D Feature Construction: Multi-view images are processed through a Transformer-based backbone to extract 2D features and build per-view cost volumes via plane sweeping. A depth module then predicts dense per-pixel depth maps to serve as geometric priors.
- Lifting and Voxelization: Using the predicted depth maps and camera intrinsics/extrinsics, the 2D image features are unprojected ("lifted") into a global 3D world space, forming an unstructured dense point cloud. This point cloud is then voxelized into a structured, sparse 3D voxel grid. Features falling within the same voxel are aggregated using average pooling.
- Sparse 3D Decoder Refinement: To enforce spatial consistency, the sparse voxel features are passed through a 3D U-Net decoder. The network predicts residual voxel features, which are added back to the original features. This residual design stabilizes training and helps capture fine-grained local details alongside broader spatial context.
- Voxel-Aligned Gaussian Prediction: Instead of mapping pixels to Gaussians, VolSplat directly regresses 3D Gaussian parameters (center offset, opacity, covariance, and spherical harmonic color coefficients) from the occupied cells of the refined 3D voxel grid.
Datasets
The authors evaluated VolSplat on three widely recognized benchmarks:
- RealEstate10K: A large-scale dataset of indoor and outdoor real-estate walkthrough videos (67,477 training scenes, 7,289 test scenes).
- ScanNet: An indoor RGB-D video dataset containing diverse room layouts (approx. 100 scenes for training, 8 for evaluation).
- ACID: An outdoor natural landscape dataset. Crucially, the authors used ACID strictly for zero-shot cross-dataset generalization testing (evaluating models trained on RealEstate10K without any fine-tuning) to demonstrate the robustness of their framework.
Key Insights and Ideas
- Volumetric Aggregation as a Filter: Fusing multi-view features into a shared 3D voxel container before predicting Gaussians naturally resolves spatial ambiguities. It significantly reduces floaters because information must be view-consistent to survive the volumetric aggregation.
- Adaptive Representation Complexity: By predicting primitives based on voxel occupancy rather than pixel grids, VolSplat naturally allocates a higher concentration of Gaussians to intricate geometric structures (e.g., object boundaries) while keeping flat or empty spaces sparse.
- Eliminating Ad-Hoc Priors: Unlike some recent hybrid methods, VolSplat's unified voxel space allows a standard 3D U-Net to reason about geometry and appearance jointly, eliminating the need for auxiliary supervision signals like ground-truth point clouds during training.
Key Results
VolSplat achieves state-of-the-art (SOTA) performance across quantitative metrics (PSNR, SSIM, LPIPS) while maintaining high computational efficiency.
Quantitative Performance
- RealEstate10K (6-view setting): VolSplat achieves a PSNR of 31.30 dB and an SSIM of 0.941, noticeably outperforming leading pixel-aligned baselines like DepthSplat (30.52 dB / 0.931) and MVSplat (29.13 dB / 0.924).
- Zero-Shot Generalization (ACID): When evaluated on outdoor scenes after being trained only on indoor data, VolSplat demonstrates remarkable robustness, hitting 32.65 dB PSNR, compared to DepthSplat's 28.37 dB and MVSplat's 28.15 dB.
Efficiency and Footprint
Despite incorporating explicit 3D feature processing, VolSplat remains highly practical for consumer-grade hardware:
- VRAM Consumption: It requires only 4.65 GB of VRAM during inference, which is significantly lower than pixelSplat (36.82 GB) and DepthSplat (8.00 GB).
- Inference Speed: It maintains a competitive latency of 0.575 seconds, proving that the sparse 3D voxel architecture introduces negligible overhead relative to its massive performance gains.
Limitations and Future Work
The primary limitation of VolSplat is its static scene assumption. Because the multi-view feature aggregation and cost-volume construction rely strictly on multi-view geometric consistency, the model struggles with dynamic objects or changing environments. Moving elements in a scene can break this consistency, resulting in rendering artifacts such as ghosting or severe blurring in the synthesized novel views.
The authors suggest that extending this voxel-aligned paradigm to handle dynamic scenes and temporal consistency represents a promising direction for future work.
Summary updated