4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

Christopher B. Choy, JunYoung Gwak, Silvio Savarese

Abstract

In many robotics and VR/AR applications, 3D-videos are readily-available input sources (a sequence of depth images, or LIDAR scans). However, in many cases, the 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose generalized sparse convolutions that encompass all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and trilateral-stationary conditional random fields that enforce spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that a convolutional neural network with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise and outperform the 3D convolutional neural network.

Introduction

In fields like robotics, autonomous driving, and AR/VR, 3D video data—sequences of depth maps or LIDAR scans—is abundant. However, most current AI models process these sequences "frame-by-frame" using 2D or 3D convolutions. This approach fails to capture the rich temporal context that connects consecutive frames.

The primary obstacle to "true" 4D perception (3D space + 1D time) is the curse of dimensionality. In a standard dense grid, adding a fourth dimension causes the number of parameters and computational cost to explode exponentially. Furthermore, 3D data is inherently sparse, making dense computations highly inefficient. Minkowski Convolutional Neural Networks address this by leveraging sparse tensors and high-dimensional convolutions.

Methodology: Generalized Sparse Convolutions

The paper introduces Generalized Sparse Convolutions, which operate only on non-empty voxels. Unlike standard sparse convolutions, this framework allows for arbitrary kernel shapes and different input/output coordinate systems.

A 4D hypercube kernel representation illustrating the high-dimensional nature of the proposed convolutions.

To manage the complexity of 4D space, the authors propose Hybrid Kernels. Instead of using a massive 4D hypercubic kernel, they employ a "cube" for spatial dimensions and a "cross" for the temporal dimension. This captures spatial geometry accurately while efficiently linking points across time with minimal parameter overhead.

The entire framework is supported by the Minkowski Engine, an open-source auto-differentiation library designed for sparse tensors. It enables the construction of deep networks in high-dimensional spaces, including 7D spaces for trilateral-stationary Conditional Random Fields (CRFs) that enforce consistency across space, time, and color.

Datasets and Spatio-Temporal Perception

The authors evaluated their work on major 3D benchmarks like ScanNet and S3DIS, as well as 4D benchmarks like RueMonge 2014. To specifically test 4D perception, they introduced Synthia 4D, a dataset of temporal driving sequences with added noise (e.g., Gaussian noise, elastic distortion).

A sequence of 3D frames from a spatio-temporal dataset, demonstrating the type of 4D data processed by MinkowskiNet.

Key Results and Insights

  • State-of-the-Art Performance: MinkowskiNet achieved 67.9% mIoU on the ScanNet benchmark, outperforming previous state-of-the-art methods by 19% at the time of publication.
  • Noise Robustness: One of the most significant findings is that 4D networks are far more robust to sensor noise than 3D counterparts. By utilizing temporal context, the network can "see through" noise in individual frames by aggregating information from adjacent frames.
  • Efficiency: Despite the high dimensionality, 4D networks using hybrid kernels added less than 6.4% more parameters compared to 3D versions.
  • Architecture Portability: The generalized sparse convolution allows for the direct porting of successful 2D architectures, such as ResNets and U-Nets, into 4D space.

Conclusion

Minkowski Convolutional Neural Networks demonstrate that sparsity is a powerful feature for high-dimensional perception. By treating 3D video as a sparse 4D tensor, the Minkowski Engine enables deep, efficient, and robust spatio-temporal modeling. While challenges remain in memory management for extremely dense environments and the need for larger 4D datasets, this work sets a new standard for 3D and 4D semantic perception.

Summary updated

Abstract

In many robotics and VR/AR applications, 3D-videos are readily-available input sources (a sequence of depth images, or LIDAR scans). However, in many cases, the 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose generalized sparse convolutions that encompass all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and trilateral-stationary conditional random fields that enforce spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that a convolutional neural network with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise and outperform the 3D convolutional neural network.
BibTeX
@inproceedings{choy2019d,
  author = {Christopher B. Choy and JunYoung Gwak and Silvio Savarese},
  title = {{4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=9JnkPIl4FE},
}
Figure