Interpolated SelectionConv for Spherical Images and Surfaces

David Hart, Michael Whitney, Bryan MorseWACV 2023

TL;DR

This paper introduces "Interpolated SelectionConv," a novel framework that allows standard 2D Convolutional Neural Networks (CNNs) to operate effectively on spherical images and 3D surfaces. By representing surfaces as graphs and interpolating convolutional filters, the method bridges the gap between specialized spherical CNNs and traditional 2D approaches, enabling efficient transfer learning and improved performance on tasks like segmentation and stylization.

What Problem Does This Paper Address?

Traditional 2D CNNs excel at processing planar images but struggle with omnidirectional (spherical) data due to inherent distortions when projecting spheres onto a 2D plane. While specialized spherical CNNs exist, they often require spherical-specific training data, which is scarce, or rely on fixed sampling strategies like the icosphere, limiting flexibility. Existing methods for adapting 2D CNNs to spherical data often involve complex transformations or suffer from artifacts, especially at seams. This paper aims to provide a general, efficient, and flexible framework for applying 2D CNNs to spherical images and general surfaces without these limitations.

Methods

The core of the proposed method is an extension of "SelectionConv," a graph-based approach for applying CNNs to irregular image types. The key innovation is Interpolated SelectionConv, which addresses the limitations of discrete selections in the original SelectionConv.

Here's a breakdown of the key techniques:

  • Graph Representation: The spherical surface is represented as a graph where nodes are points on the sphere. This representation is flexible and doesn't rely on a specific sampling strategy.
  • Selection Function: A novel selection function is introduced that operates on a local planar approximation of the sphere's surface. This allows for orientation-aware convolutions.
  • Interpolation: Instead of assigning each edge to a single discrete selection (e.g., cardinal directions), the method interpolates between multiple selections. This is achieved by assigning edge weights based on the angles between the edge's spatial relationship and the unit vectors of different selection directions. Two interpolation schemes are proposed:
  • Angle-based Interpolation: Uses angles between an edge and unit vectors of two closest selections.
  • Barycentric Interpolation: Accounts for distance from the source node, triangulating between three selections.
  • Customizable Sampling: The framework supports various sampling strategies (e.g., Fibonacci Spiral, Icosphere, Layering) for generating the graph nodes. This allows for control over the resolution and density of points on the sphere, enabling better matching to the resolution of training data.
  • Surface Generalization: The approach is extended to general 3D meshes by sampling points on the surface, storing face normals, and using a KNN approach for edge connections.

Datasets

The paper evaluates its method on the Stanford 2D-3D-S dataset, which consists of equirectangular images with 13 different semantic classes. This dataset is used for the spherical segmentation task. For mesh stylization, publicly available 3D meshes are used.

Key Insights and Ideas

  • Bridging the 2D-3D Gap: The interpolated SelectionConv effectively bridges the gap between 2D CNNs and spherical/surface data by creating a flexible graph representation that allows direct transfer of learned weights.
  • Smoothness is Key: Interpolating selection weights, rather than using discrete assignments, significantly reduces artifacts, especially along seams in spherical images and texture maps of 3D meshes. This leads to smoother and more plausible results.
  • Flexibility in Sampling: The ability to use customizable sampling strategies (like Layering) is crucial. It allows the graph resolution to be tailored to match the resolution of the original 2D training data, leading to improved performance without extensive fine-tuning.
  • Generalizability to Surfaces: The framework's extension to 3D meshes demonstrates its broad applicability beyond spheres, opening doors for various surface-based computer vision tasks.

Key Results

  • Spherical Stylization: The method produces significantly smoother and artifact-free stylizations of spherical images compared to naive approaches and the original SelectionConv, as visualized in Figure 5.
  • Spherical Segmentation: On the Stanford 2D-3D-S dataset, the proposed method achieves competitive mean Intersection-over-Union (mIOU) scores, comparable to state-of-the-art methods. Notably, it performs well even with direct transfer of weights from a 2D CNN, without extensive fine-tuning. For instance, it achieves 39.9% mIOU with direct transfer and 41.4% after fine-tuning, as shown in Table 1.
  • Mesh Stylization: The approach successfully stylizes 3D meshes with greater consistency along UV seams compared to previous methods, as illustrated in Figure 6.
  • Ablation Study: The ablation study (Table 2) highlights that the "Layering" sampling and clustering strategy, combined with angle-based interpolation, generally yields the best segmentation performance.

Limitations and Future Work

  • Memory Usage: A significant limitation is the increased memory requirement for storing graph edges and interpolation weights, especially when allowing multiple selections per edge. This can necessitate smaller batch sizes for fine-tuning.
  • Kernel Size: Similar to other methods, the edge hopping process in the convolution is not differentiable, limiting the use of larger kernel sizes (only 3x3 and 1x1 were used).
  • Future Work: The authors suggest exploring the framework's application to a wider range of general surfaces and 3D meshes. Investigating additional interpolation schemes and methods for determining expected radial distances is also proposed.

Summary updated

BibTeX
@inproceedings{hart2023interpolated,
  author = {David Hart and Michael Whitney and Bryan Morse},
  title = {{Interpolated SelectionConv for Spherical Images and Surfaces}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure