Some newer 3D reconstruction systems build a scene from a handful of photos in one network pass, with no per-scene optimization. They work by lifting image features into a 3D grid of small cubes ("voxels") and running a 3D convolutional network over that grid. The grid is the bottleneck. Its cost grows with the cube of resolution, and most of it is empty air. Two remedies exist, and nobody has combined them. The first is to compute only on occupied voxels, as VolSplat does with a sparse 3D U-Net. The second is to make the network's numbers one bit wide, as binary neural networks do. We propose to treat both remedies as the same kind of problem: a continuous quantity that must be pushed to one of two values during training. Each weight's sign must settle at −1 or +1, and each voxel must be either kept or dropped. We handle both with a single family of "bipolar" penalties: loss terms with two minima, rather than the usual weight decay that pulls everything toward zero.
The two-sentence version is this. The cost of a voxel-based reconstructor is roughly (number of active voxels) × (feature width) × (bits per number). Prior work shrinks only the first factor on sparse grids, or only the last factor on dense grids. We show that double-well penalties can shrink both at once in a feed-forward Gaussian-splatting network without collapsing rendering quality. We also test a specific, falsifiable hypothesis: on a sparse grid, a 3×3×3 binary kernel behaves more like a small kernel. If true, this defuses a known objection to binarizing volumetric convolutions.
Why this intersection, and what the evidence says
Feed-forward reconstruction has moved toward explicit 3D representations. pixelSplat predicts 3D Gaussian primitives (small colored blobs rendered by rasterization) per pixel from image pairs. FreeSplat builds cost volumes from nearby views and merges redundant Gaussians. VolSplat moves prediction into a voxel grid processed by a sparse 3D U-Net. All three use full-precision weights. FreeSplat also reports exceeding 40GB of GPU memory beyond about 50 input images. That is the volumetric memory bottleneck in its sharpest form, and it motivates our work. Coarse-to-fine designs such as CVP-MVSNet reduce how many depth hypotheses are stored. They do not change the cost per stored element, and they do not exploit emptiness.
The binarization literature supplies the method and also constrains it. Tang et al. and Liu et al. show that ordinary weight decay pulls the hidden real-valued weights behind each binary weight toward zero. Zero is exactly where their signs flip most often, so Liu et al. turn decay off entirely when weights are fully binarized. Helwegen et al. argue more broadly that continuous regularizers do not transfer to binary networks. Several papers replace decay with two-minimum penalties:
- Georges et al. use a two-Gaussian prior.
- POEM fits a two-peak mixture by expectation-maximization on point-cloud networks.
- Foothill uses learnable poles ±μ.
- BHViT uses an L1 pull toward ±1 late in training to suppress oscillation.
Sahbi's phase-field model applies a double-well penalty with minima at 0 and 1 to structural masks rather than signs. It keeps 75.38% accuracy on SBU at 99% pruning, where magnitude pruning falls to 61.53%. That result is our main reason to believe the same penalty can decide voxel existence crisply. None of these works touches volumetric reconstruction.
The closest constraint comes from BDC-Occ. It binarizes the 3D modules of a dense occupancy network and argues that binarization error grows with kernel size. It therefore relies mostly on 1×1 convolutions and keeps the image backbone in full precision. It reaches 37.24 mIoU on Occ3D-nuScenes against 37.84 for its full-precision parent. This seems to contradict our wish to keep 3×3×3 kernels, which reconstruction needs for neighborhood context. Spatial group convolution shows what happens when that context is severed: thin structures break apart. We address the conflict directly in the hypothesis below.
Core Claim
The training objective adds two double-well terms to the rendering loss of a VolSplat-style network:
$$ \mathcal{L} = \mathcal{L}_{\text{render}} + \lambda_w \sum_{i} \big(|\tilde w_i| - \mu_{\ell(i)}\big)^2 + \lambda_g \sum_{v} g_v^2 (1 - g_v)^2 + \beta \Big(\tfrac{1}{|V|}\sum_v g_v - \rho\Big)_+ . $$
Each term is defined as follows:
- $\tilde w_i$ is the hidden real-valued weight behind binary weight $\mathrm{sign}(\tilde w_i)$.
- $\mu_{\ell(i)}$ is a learnable pole magnitude for that weight's layer, as in Foothill.
- $g_v \in [0,1]$ is a gate predicted for voxel $v$ at each U-Net level. It multiplies the voxel's features, and voxels with $g_v$ below 0.5 are removed from the sparse set for all subsequent layers.
- $V$ is the current active voxel set.
- $\rho$ is a target keep-rate, and $\beta$ penalizes exceeding it.
The first penalty keeps hidden weights away from zero, where signs flip. The second pushes gates to exactly 0 or 1, so the hard pruning used at inference matches the soft gating used in training.
Our hypothesis about kernels is concrete. BDC-Occ's argument concerns a dense kernel, where all 27 neighbors contribute. In a sparse convolution on a surface-like voxel set, only the occupied neighbors contribute. Typically this is a planar subset of roughly nine. Binarization error in a sum of $k$ terms grows with $k$; the dense-kernel argument is this fan-in argument. So the error of a sparse 3×3×3 binary kernel should track its occupied fan-in, not 27. We predict that a binarized sparse U-Net can keep 3×3×3 kernels at a cost comparable to dense 1×1-dominated designs. We also predict that the gate penalty, by thinning the voxel shell, further lowers effective fan-in. The interaction is the scientific payoff: sparsity and binarization help each other rather than merely adding up.
Research Program
We start from VolSplat's architecture and keep its 2D image encoder in full precision, as both BDC-Occ and BHViT advise. Into the sparse U-Net we introduce three things: binary weights, a learnable scale per output channel, and gate heads at each resolution level. We binarize activations only inside the U-Net's middle blocks, and only in a second configuration, because the Gaussian parameter heads regress continuous color and shape. Training follows Liu et al.'s two-step schedule. First we train with binary activations and full-precision weights. Then we fully binarize, with weight decay replaced by the bipolar term rather than simply switched off. Gates are annealed from soft to hard over the second stage.
We will log three things throughout training: the sign-flip ratio, the fraction of gates in the ambiguous band, and the occupied fan-in histogram of every sparse kernel. These let us test the mechanism directly, not just the final score. Peak GPU memory is reported per stage and broken into activations versus parameters. That breakdown matters because weight binarization alone mostly saves parameter storage, while gates save activation memory. Our claims about memory will be scoped accordingly.
Baselines
The reference point is VolSplat itself, trained at full precision with no gates, which serves as the "do nothing" model. VolSplat reports PSNR 31.30, SSIM 0.941 and LPIPS 0.075 on RealEstate10K with 6 views, at 0.575s and 4.65GB on an H20 GPU (VolSplat). It is the strongest competitor because it already exploits sparsity in exactly this pipeline. We retrain it in our codebase on the same data and schedule so that differences are not artifacts of reproduction. Its paper numbers are reported alongside our reproduction. On ScanNet, FreeSplat's reported 2-view PSNR of 28.08 (FreeSplat) is the full-precision cost-volume reference. No ScanNet number for VolSplat was found on Lacuna, so we will produce one.
The strongest existing binarization competitor is a BDC-Occ-style transplant. It replaces the sparse U-Net's convolutions with BDC units dominated by 1×1 kernels plus BDC-Occ's per-channel refinement branch, trained by plain straight-through gradients. BDC-Occ's only reported number is on Occ3D-nuScenes, at 37.24 mIoU against 37.84 for full precision (BDC-Occ). No reported result was found on Lacuna for BDC-Occ on RealEstate10K or ScanNet, so we run it ourselves.
All methods share the same encoder, data, number of views, training steps, and hyperparameter search budget, which is a fixed number of trials per method. Beating the baseline means two things at matched or lower total bit-operations and peak memory. First, our model must come within 0.5 dB PSNR of reproduced full-precision VolSplat. Second, it must beat the BDC-style transplant on PSNR and on thin-structure depth error.
Three ablations isolate the key ingredient. The first trains the same binary sparse U-Net with no bipolar term: straight-through gradients with weight decay off, as Liu et al. prescribe. The second uses ordinary L2 decay. The third keeps the weight penalty but drops the gate penalty, so pruning uses plain sigmoid gates thresholded at inference.
Why not just…?
Why not just threshold VolSplat's own occupancy or opacity predictions to prune voxels? This is the most serious threat to the gate half of the proposal. VolSplat already decides occupancy from unprojected depth, and a top-k or confidence threshold costs nothing to try. Our argument is that a threshold applied after training creates a mismatch between soft training and hard inference. The phase-field results suggest this mismatch is what destroys accuracy at high sparsity: 75.38% for phase-field against 61.53% for magnitude pruning on SBU at 99% (Sahbi). That evidence comes from graph networks, not voxels. We therefore compare against VolSplat with per-level confidence thresholding at exactly matched active-voxel counts, sweeping the keep-rate. If thresholding matches our gates on the PSNR–voxel-count curve, we drop the gate penalty and say so.
Why not just quantize to 4 or 8 bits instead of 1? For photometric regression this may well win on quality per bit. The binary literature's gains are mostly measured on classification. We will train 4-bit and 8-bit quantization-aware versions of the same sparse U-Net under the same budget and plot PSNR against weight-bits × active-voxels. If 4-bit dominates the 1-bit model everywhere on that curve, the weight-binarization claim fails. The gate and fan-in findings would still stand, and we would report them that way.
Why not just make the full-precision U-Net narrower? A thinner float network at the same memory is the cheap answer to "compress the volume." We include width-reduced full-precision VolSplat variants matched on parameter bytes and on peak activation memory. Binarization is justified only if it beats them at equal storage.
Why not just use coarse-to-fine pyramids or CPU offloading? Pyramids such as CVP-MVSNet's reduce how many hypotheses are stored, but not the precision or emptiness of what is stored; they are complementary rather than competing. CLM offloads to CPU memory in per-scene optimization. It relies on each view seeing only about 0.39–1.06% of Gaussians, which is a test-time access pattern a single feed-forward pass does not have. We do not run CLM as a baseline. We do report whether our gated model's peak memory stays on-GPU in the regime where FreeSplat exceeds 40GB.
Proposed Experiments
The main benchmark is RealEstate10K at VolSplat's 6-view protocol, reporting PSNR, SSIM, LPIPS, latency, peak memory, and bit-operations. ScanNet tests indoor scenes and, crucially, view scaling. We increase input views from 2 to beyond 50 and plot peak memory and PSNR for full-precision VolSplat, threshold pruning, and our model. The prediction is that gated models grow sub-linearly in active voxels while ungated ones track FreeSplat's blow-up.
The decisive mechanistic check is the fan-in test. We fix a trained binary network and measure the error between binary and full-precision outputs per sparse 3×3×3 kernel, then regress that error on the kernel's occupied fan-in. We also compare a 3×3×3 sparse binary U-Net against a 1×1-dominated one. If error tracks occupied fan-in and the 3×3×3 version wins on thin-structure depth error, BDC-Occ's kernel-size objection does not transfer to sparse grids. If error tracks the nominal 27 and the 1×1 design wins, our hypothesis is refuted.
A second check is the sign-flip ratio under each regularizer. If the bipolar term yields better PSNR but no lower flip ratio than straight-through gradients alone, the gain is not coming from the mechanism we claim. A last sanity check retrains on DTU depth for geometric accuracy. CVP-MVSNet's reported 0.351mm overall is the full-precision reference there. This confirms that rendering gains do not hide degraded geometry.
References
- VolSplat: Rethinking Feed-Forward 3D Gaussian Splatting with Voxel-Aligned Prediction
- PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction
- FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes
- Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
- How to Train a Compact Binary Neural Network with High Accuracy?
- How Do Adam and Training Strategies Help BNNs Optimization
- Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
- Ultra-Compact NLU: Neuronal Network Binarization as Regularization
- POEM: 1-bit Point-wise Operations based on Expectation-Maximization for Efficient Point Cloud Processing
- Foothill: A Quasiconvex Regularization for Edge Computing of Deep Neural Networks
- BHViT: Binarized Hybrid Vision Transformer
- Phase-field Models for Lightweight Graph Convolutional Networks
- BDC-Occ: Binarized Deep Convolution Unit For Binarized Occupancy Network
- Efficient Semantic Scene Completion Network with Spatial Group Convolution
- CLM: Removing the GPU Memory Barrier for 3D Gaussian Splatting