Abstract
Accurate understanding of the environment is crucial for autonomous driving and robot automation. Depth sensors, including light detection and ranging (LiDAR) and depth cameras, are increasingly common, providing data in the form of depth or range images. While deep neural networks have significantly advanced segmentation tasks, most models do not explicitly account for the scale variations inherent in 3-dimensional to 2-dimensional projections. We propose a novel scale-equivariant convolution method (REconv) that utilizes the relationship between object distance and scale ratio to achieve consistent feature extraction across varying distances.
Introduction: The Challenge of Scale in Depth Images
In computer vision, 3D scenes are projected onto 2D image planes, leading to significant scale variation: objects appear smaller as their distance from the sensor increases. Standard Convolutional Neural Networks (CNNs) are inherently shift-equivariant but lack scale-equivariance. To compensate, traditional approaches rely on extensive data augmentation or high parameter counts to learn object representations at multiple scales.
Given that depth images provide explicit distance information, we can mathematically adjust convolution filters to perceive objects at a consistent scale regardless of their distance. This approach, termed Range-Equivariant Convolution (REconv), aims to "bake" scale-invariance into the network architecture itself.
Methodology: Range-Equivariant Convolution (REconv)
The core of our contribution is the REconv layer, which replaces standard convolution with a mathematically grounded alternative.
Partial Differential Operators (PDOs)
Instead of learning arbitrary weights, REconv filters are defined as linear combinations of specific differential operators, such as horizontal, vertical, and diagonal gradients.
Range-Weighting and Scale Invariance
The scale ratio in a projected image is inversely proportional to the distance ($r$). By applying a range-dependent weight $r_*$ based on pixel depth, the convolution output becomes invariant to the object's scale. To maintain computational efficiency, we implemented SREconv, which focuses on zeroth and first-order derivatives. Higher-order features are captured by stacking these layers.
Optimization and Context
A naive implementation of range-weighted convolution can be computationally expensive. We optimized this by grouping operators of the same order, resulting in a network that is only approximately 6% slower than standard baselines. Furthermore, we utilize an "Inputcat" strategy, where raw depth values are concatenated back into feature maps in later layers to preserve absolute distance context (e.g., knowing that "road" pixels typically appear at specific distances).
Experimental Evaluation
We evaluated REconv on two primary benchmarks representing different environments:
- Semantic KITTI: A large-scale outdoor dataset for autonomous driving, utilizing LiDAR point clouds projected into range images.
- NYUDv2: An indoor RGB-D dataset featuring various domestic and office scenes.
Results and Discussion
Our experiments demonstrate that REconv provides consistent improvements in segmentation accuracy with negligible increases in parameter count or execution time.
Performance Gains
- Outdoor (Semantic KITTI): REconvNet21 achieved a 1.1% increase in mean Intersection over Union (mIoU) compared to the RangeNet21++ baseline.
- Indoor (NYUDv2): Applying REconv to a DeepLabV3+ model (ResNet18 backbone) resulted in a 2.5% mIoU improvement for the 13-class segmentation task.
- Efficiency: On a GeForce 1080, the execution time for REconvNet21 was 71.5ms, compared to 67.1ms for the baseline, representing a minimal overhead for the performance gain.
Analysis of Object Scale
The most significant improvements were observed in larger objects such as cars, buildings, and furniture. These objects occupy larger image areas where scale-dependent gradients are most prominent. However, we noted a slight performance decrease for very thin objects (e.g., poles or signs), as the scale-compensation math can occasionally blur the sharp edges these objects require for identification.
Conclusion
REconv offers a robust, mathematically principled method for achieving scale-equivariance in depth image segmentation. By leveraging the physics of perspective projection, we reduce the need for brute-force data augmentation and improve the model's ability to generalize across different distances. Future work will explore integrating REconv into deeper network layers and extending the distance-compensation logic to other sensor-fusion modalities.
Summary updated