Abstract
Robotics and AR/VR applications increasingly rely on "3D videos"—sequences of depth images or LiDAR scans. Traditional processing methods often treat these frame-by-frame, missing critical temporal context. We introduce Minkowski Convolutional Neural Networks (MinkowskiNet), which utilize generalized sparse convolutions to process 4D spatio-temporal data directly. By treating time as a fourth spatial dimension and employing a novel hybrid kernel, MinkowskiNet achieves state-of-the-art performance in 3D semantic segmentation with high efficiency and robustness to noise.
The Challenge of High-Dimensional Perception
The primary obstacle in processing 3D videos is the "curse of dimensionality." Moving from 2D to 3D and eventually 4D leads to an exponential increase in parameters and computational costs. For instance, a $5 \times 5$ 2D kernel has 25 weights, whereas a 4D "tesseract" kernel of the same size requires 625 weights.
Furthermore, 3D and 4D data are inherently sparse. Traditional dense convolutions are computationally wasteful in these environments because they perform calculations on empty space.
Generalized Sparse Convolutions and the Minkowski Engine
To address these challenges, we propose the Generalized Sparse Convolution. Unlike dense convolutions that slide over every pixel, sparse convolutions only compute outputs for coordinates where data exists. This mathematical framework encompasses all discrete convolutions and is the foundation of our approach.
To implement these networks, we developed the Minkowski Engine, an open-source auto-differentiation library designed for sparse tensors. It enables efficient high-dimensional convolutions on GPUs. To further optimize 4D perception, we introduced Hybrid Kernels, which combine a 3D cubic kernel for spatial geometry with a cross-shaped temporal kernel. This design reduces memory usage and improves performance compared to full 4D tesseract kernels.
Spatio-Temporal Robustness and Results
MinkowskiNet treats time as just another spatial axis, allowing the model to learn 4D features directly. This approach proves more effective for sequence modeling than complex Recurrent Neural Networks (RNNs).
Our experiments on benchmarks like ScanNet and S3DIS demonstrate the power of this approach. MinkowskiNet achieved 73.4% mIoU on ScanNet, a 19% improvement over previous state-of-the-art results. Additionally, 4D networks show significant robustness to sensor noise; by analyzing multiple frames simultaneously, the model can maintain accuracy despite temporary occlusions or noise that would disrupt a single-frame 3D model.
Conclusion
MinkowskiNet demonstrates that high-dimensional perception is not only possible but highly efficient when leveraging sparsity. By unifying spatial and temporal dimensions into a single 4D sparse representation, we provide a robust framework for the next generation of spatio-temporal perception in robotics and beyond.
Summary updated