Neural networks for 3D scans increasingly use a hybrid design. A "point branch" applies small per-point neural layers (multi-layer perceptrons, or MLPs) to every raw 3D point. A "voxel branch" snaps the points onto a sparse 3D grid of cubes, runs 3D convolutions to gather context from neighbors, and then interpolates the result back onto the points. Pruning, meaning zeroing out redundant weights or skipping empty regions, promises large savings on paper. On GPUs those savings mostly vanish. When weights are removed at scattered positions, the cost of tracking and skipping the zeros eats the arithmetic saved. The clearest 3D example is Putting 3D Spatially Sparse Networks on a Diet, which removed 99% of a voxel network's parameters and 95% of its arithmetic yet sped inference up by only 45%.
Our claim is that hybrid networks offer a way out that voxel-only networks do not, because their two branches have opposite hardware profiles. The voxel branch already pays for irregularity in its memory access: hash lookups, gathering neighbor features, and scattering results back. Its cost scales with the number of active voxels, not with the number of weights. The point branch is a plain matrix multiply over roughly a hundred thousand points. That is the large, contiguous workload where hardware-supported structured sparsity actually delivers speed. We propose to compress each branch with the kind of sparsity its hardware rewards. Weights in the point branch get pruned in the 2:4 pattern that GPU Tensor Cores execute natively. The voxel branch gets pruned spatially, by dropping active voxels. Points whose voxels are dropped skip the point-to-voxel conversion entirely, because the point branch already carries their features. We allocate the budget between branches with a measured latency model rather than a count of arithmetic operations.
Core Claim
In a point-voxel network, end-to-end GPU latency falls furthest per unit of accuracy lost under three rules. Fixed-pattern 2:4 weight sparsity goes only into the point-wise MLPs. The voxel branch is thinned by removing active voxels rather than weights. Points in removed regions bypass both voxelization (point-to-grid) and devoxelization (grid-to-point). The novelty is not "prune a 3D network," which has been done. It is the argument that the hybrid structure itself determines where each kind of sparsity can turn into wall-clock speed. It is also the use of the point branch as a fallback that makes aggressive spatial pruning of the voxel branch safe.
The argument draws on three constraints from the literature. The first is that unstructured weight sparsity does not pay inside sparse 3D convolutions. Sparse GPU Kernels for Deep Learning built custom kernels that beat dense code from about 71% sparsity, but only on long, uniform matrix rows. TorchSparse: Efficient Point Cloud Inference Engine shows that a sparse 3D convolution breaks into many small per-offset matrix multiplies and spends up to half its time on gathering and scattering. Weight sparsity shortens none of that data movement. Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity makes the general point that the bookkeeping for scattered zeros often exceeds the savings.
The second constraint is that 2:4 sparsity is safe and fast on large dense multiplies. Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch trains such networks stably with its sparse-refined straight-through estimator. It reports 77.0% top-1 accuracy for ResNet-50 at 2:4 against a 77.3% dense baseline. It also notes that the pattern gives no speedup without Ampere-class hardware.
The third constraint is that spatial pruning works and is cheap. Spatial Pruned Sparse Convolution for Efficient 3D Object Detection drops low-magnitude voxels and reports a 52.4% reduction in giga-floating-point operations (GFLOPs) on KITTI with accuracy within 0.1 to 0.2% of the unpruned model on nuScenes. That work studies voxel-only backbones, where a dropped voxel simply passes its features through unchanged via an identity shortcut. In a hybrid network a dropped voxel is still represented by its points, which is why we expect hybrids to tolerate much heavier spatial pruning.
Relation to Prior Work
The closest work is Putting 3D Spatially Sparse Networks on a Diet. It applies global magnitude weight pruning to a voxel-only sparse U-Net, executes the result with compressed-sparse-row kernels, and finds that weights along the vertical axis survive pruning disproportionately. We differ in three ways. We study hybrid networks. We deliberately keep voxel-branch weights dense, on the hypothesis that weight sparsity there cannot be turned into speed. And we target a hardware-native sparsity pattern rather than a compressed-index format. Their 45% speedup from a 95% arithmetic cut is the motivating evidence for our approach.
The hybrid architectures we build on are Point-Voxel CNN for Efficient 3D Deep Learning, which introduced the two-branch design, and Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution, which made the voxel branch sparse, cut conversion cost to about 4% of computation with GPU hash tables, and searched channel widths and depths for latency. Both keep all weights dense and never prune voxels.
PV-RCNN is a two-stage hybrid detector; its point stages are slow for reasons outside our scope, namely sampling and neighbor search. GraVoS and TSP3D prune voxels, but GraVoS does so only offline during training and TSP3D uses text conditioning. Dual-side Sparse Tensor Core shows that skipping dynamic activation zeros needs new hardware. That constrains us to prune voxels at whole-set granularity that existing gather kernels already handle, rather than skipping individual zero activations. Tile-Wise Sparsity supplies the fallback weight pattern for GPUs without 2:4 support.
Research Program
We first profile before building anything. We will instrument SPVNAS-style models running on TorchSparse on an Ampere GPU (RTX 3090 or A100), on full SemanticKITTI scans. We will attribute latency to five components: hashing and voxelization, gather and scatter, voxel-branch matrix multiplies, point-branch MLPs, and devoxelization. The whole proposal depends on the point-branch MLPs taking a non-trivial share of latency and running as large, well-shaped multiplies. If they take under roughly 15% of time, 2:4 sparsity there cannot matter much. In that case the work reduces to spatial pruning plus conversion skipping, and we will say so.
Next we build the compressed network, which has three parts. The first is 2:4 weight sparsity in every point-wise MLP, trained or fine-tuned with the sparse-refined straight-through estimator and run through Tensor Core sparse matrix kernels. The second is spatial pruning in the voxel branch. Each sparse convolution stage keeps a top-$k$ fraction of active voxels. We will compare two scoring rules: feature magnitude, as in SPS-Conv, and a small learned score computed from the point-branch features already on hand. Using the point features avoids paying for the voxel computation in order to decide whether to skip it. The third part is conversion skipping. Points whose voxel is dropped neither scatter into the hash table at the next stage nor receive interpolated features. They continue on point-branch features alone. This addresses the problem raised by the Amdahl's-law argument in the research file: once arithmetic shrinks, the conversion cost that was about 4% would otherwise come to dominate runtime.
We set the per-stage voxel keep ratio and the choice of which MLPs receive 2:4 by fitting a per-operator latency model from the profiling runs, $T = \sum_\ell \hat t_\ell(V_\ell, N_\ell, s_\ell)$. Here $\hat t_\ell$ is the measured latency of operator $\ell$ as a function of its active voxel count $V_\ell$, its active point count $N_\ell$, and whether its weights are 2:4 sparse ($s_\ell$). We then search for the configuration that minimizes validation loss under a latency target. The search reuses the progressive shrinking machinery of SPVNAS rather than inventing a new procedure.
Baselines
The naive reference point is the dense SPVNAS model at the operating point we start from, run on TorchSparse on the same GPU. Every result is reported as a change from it in mean intersection-over-union (mIoU) and measured end-to-end latency, including hashing and conversion. The file gives no absolute SemanticKITTI mIoU for SPVNAS (no reported result found on Lacuna for SPVNAS's absolute mIoU). We will therefore retrain it ourselves and anchor against what the cited papers do report. SPVNAS reports +3.3 mIoU over MinkowskiNet while running 3× faster on a GTX 1080Ti. TorchSparse reports MinkUNet at 36 frames per second on an RTX 3090.
The strongest competitor on latency is the latency-aware architecture search of SPVNAS itself. It produces dense, narrower or shallower networks tuned for the target device, which is exactly what a reviewer would propose instead of sparsity. We will run its search with the same training budget and the same GPU, and extract models at our latency targets. We beat it only if we reach higher mIoU at equal measured latency across at least three latency targets.
The strongest existing weight-pruning method is the global magnitude pruning of WS³-ConvNet, applied to both branches of the hybrid with its compressed-sparse-row execution. Its reported result is 69.4% mIoU against a 71.6% baseline on ScanNet, with a 45% inference speedup. We will run it on ScanNet with a hybrid backbone under the same fine-tuning epochs. Beating it means a larger measured speedup at equal or smaller mIoU loss.
The strongest spatial-pruning baseline is SPS-Conv's magnitude-based voxel pruning grafted onto the SPVNAS voxel branch with dense weights. The cited result is a 52.4% GFLOPs reduction on KITTI and accuracy within 0.1 to 0.2% on nuScenes. That paper reports GFLOPs, not wall-clock time, so we will measure its latency ourselves.
Ablations isolate each ingredient. We remove 2:4 from the point branch, which leaves spatial pruning only. We remove spatial pruning, which leaves 2:4 only. We disable conversion skipping. We swap the point-feature voxel scorer for the magnitude scorer. Finally, we apply 2:4 uniformly to both branches, including the per-offset voxel convolution weights. That last ablation directly tests the central claim that voxel-branch weight sparsity does not pay.
Why not just…?
Why not just apply 2:4 sparsity uniformly to the whole network? It is the simplest option, and the research file shows it is accurate on 2D classifiers. The doubt is about speed, not accuracy. TorchSparse's own analysis shows voxel-branch multiplies are small and fragmented, and that half the convolution time is data movement that 2:4 leaves untouched. The uniform-2:4 ablation settles this directly. If it matches our method's speed at equal accuracy, the branch-asymmetric claim is wrong and we will report that.
Why not just shrink the network with SPVNAS's search? This is the real threat. A dense narrow network has zero sparsity overhead, and SPVNAS already optimizes measured latency. SPVNAS itself observes that narrowing channels helps latency less than reducing depth. That leaves room for an approach that keeps width but drops spatial work. Still, we cannot argue our way past this baseline. The matched-latency comparison against searched dense models is the experiment that decides whether the proposal is worth adopting.
Why not just use coarser voxels? Coarser voxels cut active voxels roughly with the cube of the voxel size, with no new machinery. However, SPVNAS attributes a 25% gain on motorcyclists to keeping a fine 0.05 m grid, and PVCNN documents information lost for thin structures. Our spatial pruning instead coarsens adaptively: it drops background voxels while keeping fine resolution on objects. We will include a coarse-voxel baseline at 0.10 m and 0.20 m at matched latency and report per-class mIoU on small classes, where the difference should appear.
Why not just use unstructured sparsity with a good kernel library? Sputnik shows custom kernels beat cuSPARSE by 3.58× and pass dense code at around 71% sparsity, but only on the long matrix rows of 2D networks. The 45% speedup at 99% sparsity in WS³-ConvNet suggests this does not transfer to 3D convolutions. The WS³-ConvNet baseline above tests that transfer directly.
Proposed Experiments
The main benchmark is SemanticKITTI single-scan segmentation, following the protocol of SemanticKITTI. The secondary benchmark is ScanNet, chosen so that the comparison with WS³-ConvNet happens on that paper's own dataset. We will also add nuScenes segmentation as a check that the results hold on sparser LiDAR. Metrics are mIoU, per-class IoU on small classes (motorcyclist, bicyclist, pole), and measured latency at batch size one. Latency includes all hashing and conversion steps, is reported as a median over the validation set, and is measured on an RTX 3090 and an A100.
There are four decisive checks. The first is the profiling result: the point-branch share of latency determines whether the 2:4 component can matter at all. The second is a speed-versus-accuracy curve against latency-searched dense SPVNAS models. The proposal succeeds only if it improves mIoU at matched measured latency, not matched operation counts. The third is the uniform-2:4 ablation. It should show that sparsifying voxel-branch weights adds accuracy loss without a proportional latency gain. The fourth compares spatial pruning in the hybrid against spatial pruning in a voxel-only MinkUNet at the same keep ratios. Our account predicts the hybrid loses less small-object IoU at high pruning rates, because its points carry the dropped regions. If it does not, the main structural argument of this note fails.
A final run on a pre-Ampere GPU, a GTX 1080Ti, using tile-wise rather than 2:4 point-branch sparsity, will show how much of the gain depends on that specific hardware.
References
- Putting 3D Spatially Sparse Networks on a Diet
- Sparse GPU Kernels for Deep Learning
- TorchSparse: Efficient Point Cloud Inference Engine
- Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- Spatial Pruned Sparse Convolution for Efficient 3D Object Detection
- Point-Voxel CNN for Efficient 3D Deep Learning
- Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution
- PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
- GraVoS: Voxel Selection for 3D Point-Cloud Detection
- Text-guided Sparse Voxel Pruning for Efficient 3D Visual Grounding
- Dual-side Sparse Tensor Core
- Accelerating sparse DNN models without hardware-support via tile-wise sparsity
- SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences