Abstract
Recent advancements in 3D semantic segmentation have seen a shift from traditional Sparse CNNs to Point Transformers due to the latter's superior adaptivity. However, Transformers often suffer from high computational costs and memory overhead. This paper introduces OA-CNNs (Omni-Adaptive Sparse CNNs), an architecture that integrates lightweight adaptive modules into a convolutional framework. By employing spatially adaptive receptive fields and dynamic local relation modeling, OA-CNNs achieve state-of-the-art accuracy on major benchmarks while maintaining a 5× speed advantage and significantly lower memory consumption compared to Transformer-based models.
Introduction
In 3D scene understanding, the choice of architecture typically involves a trade-off between the efficiency of Sparse CNNs and the adaptivity of Point Transformers. Sparse CNNs are highly efficient but traditionally rely on static kernels that apply the same fixed-size filters regardless of local geometry. Conversely, Point Transformers use attention mechanisms to adapt their focus based on input features, leading to high accuracy at the cost of significant latency and memory usage.
OA-CNNs bridge this gap by answering a fundamental question: Can the adaptivity of Transformers be achieved within the efficient framework of convolutions? The authors propose that the success of Transformers stems from their adaptive nature rather than the specific attention operation, and demonstrate that CNNs can reclaim performance leadership by incorporating similar flexibility.
Methodology
OA-CNNs introduce two primary components to achieve "omni-adaptivity":
1. Spatially Adaptive Receptive Fields
Different geometric structures require varying scales of context. For instance, a flat floor can be identified with a broad, low-resolution view, whereas a complex junction between a wall and ceiling requires a fine-grained, high-resolution focus. OA-CNNs utilize a Pyramid Grid Partition to observe the scene at multiple scales simultaneously. A lightweight selective aggregator then dynamically determines the optimal scale for each voxel.
2. Adaptive Relation Convolution (ARConv)
To replace expensive self-attention, the authors propose ARConv, which follows a "multi-one-multi" paradigm. Instead of calculating all-to-all relationships, ARConv computes a single "centroid" feature for a local grid. Individual voxels then determine their weights based on their specific relationship to this centroid. This approach captures complex local dependencies with linear complexity, making it significantly faster than standard attention mechanisms.
Experimental Results
The model was evaluated on several standard 3D benchmarks, including indoor scenes (ScanNet v2, S3DIS) and outdoor driving datasets (nuScenes, SemanticKITTI).
Quantitative Performance
OA-CNNs achieved a 76.1% mIoU on ScanNet v2 and 78.9% on nuScenes, outperforming top-tier Transformer models like PTv2 and OctFormer. Crucially, the model maintains a 5× speed advantage on an RTX 3090 and demonstrates a much lower memory footprint, which is vital for deployment on edge devices like autonomous vehicles and robots.
Qualitative Insights
Visualizations confirm that the model successfully adapts its receptive field size based on geometry. As shown in the figures, the network "shrinks" its focus for complex edges and junctions while "expanding" it for uniform surfaces like walls and tables, validating the effectiveness of the Pyramid Grid Partition.
Conclusion and Future Work
OA-CNNs prove that Sparse CNNs remain highly competitive for 3D perception when enhanced with adaptive modules. By matching the accuracy of Transformers while preserving the efficiency of convolutions, this architecture provides a scalable path for real-time 3D semantic segmentation. Future research may explore the use of Neural Architecture Search (NAS) to automatically optimize grid sizes, potentially further improving performance across diverse datasets.
Summary updated