Interpolation-Aware Padding for 3D Sparse Convolutional Neural Networks

Yu-Qi Yang, Peng-Shuai Wang, Yang LiuICCV 2021 · OpenReview

Abstract

Sparse 3D Convolutional Neural Networks (CNNs) are the standard for efficient point cloud processing, yet they face challenges in fine-grained tasks that require point-wise features. Because sparse grids omit "empty" space, trilinear interpolation often lacks the necessary neighboring voxels. We introduce interpolation-aware padding, a surgical technique that pads only the specific empty voxels required for interpolation. This enables the network to learn meaningful features for empty space, significantly improving performance in 3D semantic segmentation and object detection without the memory overhead of dense padding.

Introduction

In 3D deep learning, efficiency is paramount. Sparse voxel-based CNNs achieve this by performing computations only on occupied voxels. However, tasks like semantic segmentation or object detection require features at specific point locations, typically obtained via trilinear interpolation from the surrounding eight grid corners. In a sparse representation, many of these corners are empty, leading to "blocky" results from nearest-neighbor approximations or inconsistent features from zero-padding.

The Interpolation-Aware Padding Method

The core innovation is a targeted padding scheme that bridges the gap between efficiency and feature continuity. Instead of padding a full "ring" of voxels around every occupied cell—which is memory-intensive—the proposed algorithm identifies exactly which empty voxels are needed to perform mathematically complete trilinear interpolation for every input point.

The end-to-end pipeline of the proposed method, from input point cloud to final MLP-based point-wise feature extraction.

The process follows these steps:

  1. Voxelization: Mapping the input point cloud to a sparse grid.
  2. Targeted Padding: For each point, the eight nearest grid corners are identified. If a corner is empty, a "padded" voxel is instantiated.
  3. Feature Learning: These padded voxels are initialized to zero but participate in subsequent 3D convolutions. This allows the network to propagate context from occupied regions into these "buffer" zones.
  4. Interpolation: Points extract features from a complete set of eight neighbors, enabling standard trilinear interpolation.
Comparison of different padding strategies: (a) zero padding, (b) 1-ring padding, (c) octree padding, and (d) the proposed interpolation-aware padding.

As shown in the comparison above, interpolation-aware padding (d) is significantly more efficient than 1-ring padding (b) while providing better coverage than zero padding (a).

Experimental Evaluation

The method was evaluated on three major benchmarks: PartNet (fine-grained shape segmentation), ScanNet (indoor scene segmentation and detection), and KITTI (outdoor LiDAR segmentation).

Key Results

  • Accuracy Improvements: The scheme consistently outperformed baselines, improving mean Intersection over Union (mIoU) by 2.2% on PartNet, 2.0% on ScanNet, and 2.4% on KITTI.
  • Detection Gains: For 3D object detection on ScanNet, the method yielded a 2.0 point improvement in [email protected].
  • Memory Efficiency: On the KITTI dataset, targeted padding required only 6.4GB of VRAM, compared to 12.9GB for 1-ring padding—a 50% reduction that makes training on large scenes much more feasible.

Discussion and Insights

A key finding is that padding acts as an "information bridge." In high-resolution sparse grids, occupied voxels can become isolated. Padding creates pathways for features to flow across small gaps during convolution. Furthermore, the network learns to populate these padded voxels with boundary-aware features rather than leaving them at zero, providing much richer context for interpolation than a hardcoded zero.

Conclusion

Interpolation-aware padding offers a simple yet effective "plug-in" for sparse CNN architectures. By strategically adding only the necessary voxels for interpolation, it allows models to achieve higher accuracy at coarser resolutions, optimizing the trade-off between computational cost and performance.

Summary updated

Abstract

Sparse voxel-based 3D convolutional neural networks (CNNs) are widely used for various 3D vision tasks. Sparse voxel-based 3D CNNs create sparse non-empty voxels from the 3D input and perform 3D convolution operations on them only. We propose a simple yet effective padding scheme --- interpolation-aware padding to pad a few empty voxels adjacent to the non-empty voxels and involve them in the 3D CNN computation so that all neighboring voxels exist when computing point-wise features via the trilinear interpolation. For fine-grained 3D vision tasks where point-wise features are essential, like semantic segmentation and 3D detection, our network achieves higher prediction accuracy than the existing networks using the nearest neighbor interpolation or the normalized trilinear interpolation with the zero-padding or the octree-padding scheme. Through extensive comparisons on various 3D segmentation and detection tasks, we demonstrate the superiority of 3D sparse CNNs with our padding scheme in conjunction with feature interpolation.
BibTeX
@misc{yang2021interpolation,
  author = {Yu-Qi Yang and Peng-Shuai Wang and Yang Liu},
  title = {{Interpolation-Aware Padding for 3D Sparse Convolutional Neural Networks}},
  year = {2021},
  eprint = {2108.06925},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.06925},
}
Figure