SpOctA: A 3D Sparse Convolution Accelerator with Octree-Encoding-Based Map Search and Inherent Sparsity-Aware Processing

Dongxu Lyu, Zhenyu Li, Yuzhou Chen, Jinming Zhang, Ningyi Xu, Guanghui He

Abstract

3D perception is a cornerstone of modern autonomous systems, yet the computational demands of 3D Sparse Convolution (SpConv) often exceed the capabilities of general-purpose hardware. This paper introduces SpOctA, a specialized ASIC accelerator designed to address the unique bottlenecks of sparse 3D data. By leveraging a parallelized octree-based map search and a sparsity-aware computing core, SpOctA achieves significant improvements in both throughput and energy efficiency. Experimental results across major benchmarks like ScanNet and nuScenes demonstrate up to a 6.9× speedup and 3.1× better energy efficiency compared to existing state-of-the-art accelerators.

Introduction

3D perception is critical for the safe operation of autonomous vehicles, robotics, and immersive AR/VR environments. Unlike 2D images, 3D point clouds are inherently sparse; the vast majority of a 3D volume consists of empty space. To process this data efficiently, researchers utilize Sparse Convolution (SpConv), which restricts calculations to voxels containing actual data.

While SpConv reduces the theoretical workload, it introduces significant hardware challenges. Specifically, the "map search" required to identify a voxel’s neighbors is typically a serial, memory-intensive process. Furthermore, 3D kernels scale cubically ($K^3$), meaning a standard $3 \times 3 \times 3$ kernel requires 27 weights, tripling the workload of a 2D counterpart. Finally, the memory footprint for storing sparse indices and weights often exceeds the on-chip capacity of traditional edge devices. SpOctA is proposed as a comprehensive architectural solution to these three bottlenecks.

The SpOctA Architecture

SpOctA employs a dual-core architecture supported by a specialized memory management system to streamline the SpConv pipeline.

Traditionally, map search is a serial "pointer-chasing" task that creates a massive latency bottleneck. The OCTENT Core overcomes this by utilizing Octree Encoding. By representing 3D space as a hierarchical tree, the hardware can perform neighbor queries in parallel. The architecture utilizes a Parallel Neighbor Encoding Look-Up Table (PNELUT), allowing the system to evaluate multiple neighbor candidates simultaneously, transforming a search problem into a structured table-indexing operation.

SPAC Core: Sparsity-Aware Computing

Even within the sparse regions of a point cloud, further efficiency can be gained by examining the data values themselves. In many neural networks, 40–60% of activations are zero due to ReLU functions. The SPAC Core (Sparsity-Aware Computing) features a "Gather Unit" that identifies and skips these zero-value activations. By only feeding non-zero data into the Multiply-Accumulate (MAC) units, the core saves significant clock cycles and power.

Non-Uniform Caching and Pipelining

SpOctA introduces a novel Non-Uniform Caching strategy based on the physical characteristics of LiDAR sensors. LiDAR data typically exhibits higher horizontal resolution than vertical resolution, leading to an asymmetric distribution of data. The authors observed that weights associated with the middle horizontal plane ($z=0$) are accessed much more frequently. SpOctA prioritizes these "hot" weights for on-chip caching, reducing external DRAM energy consumption by 57.6%. Additionally, the architecture uses fine-grained pipelining to overlap the map search with computation, effectively hiding search latency.

Evaluation

The performance of SpOctA was evaluated using four representative 3D benchmarks:

  • ScanNet & SemanticKITTI: Indoor and outdoor semantic segmentation using MinkowskiUNet.
  • KITTI & nuScenes: Outdoor object detection using the SECOND model.

Key Results

Compared to prior specialized hardware like PointAcc, SpOctA demonstrates superior performance metrics:

  • Search Acceleration: The OCTENT core provides an 8.8× to 21.2× speedup in neighbor search tasks.
  • Throughput: Overall system performance improved by 1.1× to 6.9× across the tested benchmarks.
  • Efficiency: Energy efficiency is 1.5× to 3.1× higher than previous state-of-the-art accelerators.
  • Real-Time Performance: On the nuScenes benchmark, SpOctA achieved significantly higher normalized frame rates than high-end GPUs like the NVIDIA RTX 3090Ti, proving its viability for real-time edge deployment.

Conclusion and Future Work

SpOctA successfully addresses the primary bottlenecks of 3D Sparse Convolution by parallelizing map searches and exploiting both spatial and value-based sparsity. While the current architecture excels with standard $3 \times 3 \times 3$ kernels, future iterations may explore dilated kernels and more aggressive weight compression techniques. By optimizing for sensor-specific data distributions, SpOctA provides a blueprint for the next generation of energy-efficient 3D perception hardware.

Summary updated

BibTeX
@inproceedings{lyu2023spocta,
  author = {Dongxu Lyu and Zhenyu Li and Yuzhou Chen and Jinming Zhang and Ningyi Xu and Guanghui He},
  title = {{SpOctA: A 3D Sparse Convolution Accelerator with Octree-Encoding-Based Map Search and Inherent Sparsity-Aware Processing}},
  booktitle = {ICCAD},
  year = {2023},
  doi = {10.1109/iccad57390.2023.10323728},
}
Figure