OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation

Bohao Peng, Xiaoyang Wu, Li Jiang, Yukang Chen, Hengshuang Zhao, Zhuotao Tian, Jiaya Jia

Abstract

Recent advancements in 3D semantic segmentation have seen a shift from traditional Sparse CNNs to Point Transformers due to the latter's superior adaptivity. However, Transformers often suffer from high computational costs and memory overhead. This paper introduces OA-CNNs (Omni-Adaptive Sparse CNNs), an architecture that integrates lightweight adaptive modules into a convolutional framework. By employing spatially adaptive receptive fields and dynamic local relation modeling, OA-CNNs achieve state-of-the-art accuracy on major benchmarks while maintaining a 5× speed advantage and significantly lower memory consumption compared to Transformer-based models.

Introduction

In 3D scene understanding, the choice of architecture typically involves a trade-off between the efficiency of Sparse CNNs and the adaptivity of Point Transformers. Sparse CNNs are highly efficient but traditionally rely on static kernels that apply the same fixed-size filters regardless of local geometry. Conversely, Point Transformers use attention mechanisms to adapt their focus based on input features, leading to high accuracy at the cost of significant latency and memory usage.

OA-CNNs bridge this gap by answering a fundamental question: Can the adaptivity of Transformers be achieved within the efficient framework of convolutions? The authors propose that the success of Transformers stems from their adaptive nature rather than the specific attention operation, and demonstrate that CNNs can reclaim performance leadership by incorporating similar flexibility.

Methodology

OA-CNNs introduce two primary components to achieve "omni-adaptivity":

1. Spatially Adaptive Receptive Fields

Different geometric structures require varying scales of context. For instance, a flat floor can be identified with a broad, low-resolution view, whereas a complex junction between a wall and ceiling requires a fine-grained, high-resolution focus. OA-CNNs utilize a Pyramid Grid Partition to observe the scene at multiple scales simultaneously. A lightweight selective aggregator then dynamically determines the optimal scale for each voxel.

2. Adaptive Relation Convolution (ARConv)

To replace expensive self-attention, the authors propose ARConv, which follows a "multi-one-multi" paradigm. Instead of calculating all-to-all relationships, ARConv computes a single "centroid" feature for a local grid. Individual voxels then determine their weights based on their specific relationship to this centroid. This approach captures complex local dependencies with linear complexity, making it significantly faster than standard attention mechanisms.

Visualization of adaptive receptive fields across different objects. Complex structures like junctions use smaller, more focused fields, while uniform surfaces like walls use larger ones.

Experimental Results

The model was evaluated on several standard 3D benchmarks, including indoor scenes (ScanNet v2, S3DIS) and outdoor driving datasets (nuScenes, SemanticKITTI).

Quantitative Performance

OA-CNNs achieved a 76.1% mIoU on ScanNet v2 and 78.9% on nuScenes, outperforming top-tier Transformer models like PTv2 and OctFormer. Crucially, the model maintains a 5× speed advantage on an RTX 3090 and demonstrates a much lower memory footprint, which is vital for deployment on edge devices like autonomous vehicles and robots.

Performance comparison showing OA-CNNs (Ours) achieving higher mIoU with significantly lower latency and memory usage compared to state-of-the-art Transformers.

Qualitative Insights

Visualizations confirm that the model successfully adapts its receptive field size based on geometry. As shown in the figures, the network "shrinks" its focus for complex edges and junctions while "expanding" it for uniform surfaces like walls and tables, validating the effectiveness of the Pyramid Grid Partition.

Conclusion and Future Work

OA-CNNs prove that Sparse CNNs remain highly competitive for 3D perception when enhanced with adaptive modules. By matching the accuracy of Transformers while preserving the efficiency of convolutions, this architecture provides a scalable path for real-time 3D semantic segmentation. Future research may explore the use of Neural Architecture Search (NAS) to automatically optimize grid sizes, potentially further improving performance across diverse datasets.

Summary updated

Abstract

The booming of 3D recognition in the 2020s began with the introduction of point cloud transformers. They quickly overwhelmed sparse CNNs and became state-of-the-art models, especially in 3D semantic segmentation. However, sparse CNNs are still valuable networks, due to their efficiency treasure, and ease of application. In this work, we reexamine the design distinctions and test the limits of what a sparse CNN can achieve. We discover that the key credit to the performance difference is adaptivity. Specifically, we propose two key components, i.e., adaptive receptive fields (spatially) and adaptive relation, to bridge the gap. This exploration led to the creation of Omni-Adaptive 3D CNNs (OA-CNNs), a family of networks that integrates a lightweight module to greatly enhance the adaptivity of sparse CNNs at minimal computational cost. Without any self-attention modules, OA-CNNs favorably surpass point transformers in terms of accuracy in both indoor and outdoor scenes, with much less latency and memory cost. Notably, it achieves 76.1%, 78.9%, and 70.6% mIoU on ScanNet v2, nuScenes, and SemanticKITTI validation benchmarks respectively, while maintaining at most 5x better speed than transformer counterparts. This revelation highlights the potential of pure sparse CNNs to outperform transformer-related networks.
BibTeX
@inproceedings{peng2024oa,
  author = {Bohao Peng and Xiaoyang Wu and Li Jiang and Yukang Chen and Hengshuang Zhao and Zhuotao Tian and Jiaya Jia},
  title = {{OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure