Abstract
Standard Convolutional Neural Networks (CNNs) are optimized for grid-like data, such as images. However, many real-world datasets, including LiDAR point clouds and superpixels, are "arbitrarily structured." This paper introduces the Spatial Generalized Propagation Network (SGPN), a novel deep learning module designed to model global pairwise relationships in irregular data by representing it as a collection of Directed Acyclic Graphs (DAGs). By learning a linear diffusion process, SGPN significantly improves semantic segmentation performance across various data formats while remaining computationally efficient.
The Challenge of Irregular Data
While CNNs excel at processing pixels arranged in neat grids, they struggle with unstructured data like scattered points in a LiDAR scan. Researchers typically address this by either voxelizing the data—which wastes memory and loses detail—or using Non-Local blocks to model every point's relationship to every other point. The latter is computationally expensive and often ignores the underlying geometry of the object.
A key issue in modeling global affinities is "bleeding" information across unrelated boundaries. For instance, in a curved object, two points might be physically close in Euclidean space but belong to different parts of the structure. SGPN addresses this by ensuring propagation follows the actual "skeleton" of the data.
Spatial Generalized Propagation Network (SGPN)
The SGPN module generalizes Spatial Propagation Networks (SPNs) to any arbitrary graph. The core innovation lies in how information is propagated:
- Graph Representation: Data is represented as a set of Directed Acyclic Graphs (DAGs). For point clouds, these graphs can follow principal axes in multiple directions.
- Learnable Affinities: SGPN learns the "affinity" (similarity) between connected nodes using a differentiable kernel. This allows the network to dynamically decide how much information should flow between points based on features like color or geometry.
- Linear Diffusion: Information is propagated across the graph using a process equivalent to linear diffusion, effectively "smoothing" features across the entire object through chains of connections.
- Parallel Execution: To ensure efficiency, graph nodes are re-ordered into groups that can be processed simultaneously, preventing the module from becoming a training bottleneck.
Key Insights: Path-Awareness and Geometry
The authors emphasize that path-awareness is crucial. By propagating along the graph's structure rather than relying on Euclidean distance, SGPN avoids mixing features from distinct structural components. Furthermore, incorporating geometric hints like surface normals and XYZ coordinates into the affinity learning process helps the network understand that points on the same smooth surface should likely share semantic labels.
Experimental Results
SGPN was evaluated on several benchmarks, demonstrating significant improvements:
- Image Segmentation (Cityscapes): Integrating SGPN into a Dilated Residual Network (DRN) backbone increased the mean Intersection over Union (mIoU) from 69.5% to 75.0%, outperforming memory-intensive Non-Local modules.
- Point Cloud Segmentation (RueMonge2014): SGPN achieved an mIoU of 73.66%, a substantial jump over the previous state-of-the-art, SplatNet (69.8%).
- Versatility: The module also proved effective for tasks like color restoration and scribble-based colorization, where a single user stroke can propagate color accurately across an entire object by following learned structural paths.
Conclusion
SGPN provides a flexible and efficient way to model global relationships in non-grid data. By generalizing propagation to arbitrary graphs and emphasizing path-aware affinities, it bridges the gap between grid-based CNNs and the unstructured reality of 3D data. Future work may explore end-to-end learning of the graph topology itself, further automating the discovery of optimal information flow paths.
Summary updated