Sparse Convolutional Networks using the Permutohedral Lattice

Martin Kiefel, Varun Jampani, Peter V. GehlerarXiv 2015 · OpenReview

TL;DR

This paper introduces a novel framework for learning generalized bilateral filters, moving beyond fixed Gaussian kernels to learn arbitrary filter shapes directly from data. This is achieved by leveraging the permutohedral lattice for efficient, high-dimensional sparse convolutions, enabling end-to-end learning in CNNs and improved performance in tasks like image segmentation and denoising.

What Problem Does This Paper Address?

Traditional image filtering techniques, like the widely used bilateral filter, rely on predefined parametric forms, most commonly Gaussian kernels. While effective at preserving edges, these fixed kernels limit the filter's adaptability to complex image structures and data distributions. The challenge is to create filters that can learn optimal shapes from data, especially in high-dimensional feature spaces where data is often sparse, without sacrificing computational efficiency. This paper addresses this by enabling the learning of arbitrary, high-dimensional filters that operate efficiently on sparse data.

Methods

The core contribution is the generalization of the permutohedral lattice convolution, originally developed for approximate Gaussian bilateral filtering, to learn arbitrary filter kernels. The authors extend the framework from Adams et al. [2] to allow for non-separable filters and derive gradient descent algorithms, enabling end-to-end learning via backpropagation.

Key methodological components include:

  • Permutohedral Lattice Convolution: This technique efficiently handles high-dimensional data by projecting it onto a lattice structure. The process involves three steps:
  • Splatting: Distributing input signal values to lattice corners based on barycentric coordinates.
  • Convolution: Performing the filter operation on the lattice.
  • Slicing: Interpolating the results back to the original feature locations.
  • Learning Arbitrary Filter Kernels: By treating the filter weights on the permutohedral lattice as learnable parameters, the authors enable gradient-based optimization. This allows the filter to adapt its shape to the specific data and task.
  • High-Dimensional Sparse Feature Spaces: The permutohedral lattice is particularly well-suited for sparse data in high dimensions. The authors leverage this by defining feature spaces that can include position, color, and other modalities, allowing filters to capture complex relationships.
  • Bilateral Neural Networks (BNNs): The learned bilateral filters are integrated as "bilateral convolution layers" (BCLs) within Convolutional Neural Networks (CNNs). This allows for end-to-end training of networks that can process high-dimensional sparse data and adapt their receptive fields based on image content.

Datasets

The paper evaluates its methods on a variety of standard datasets:

  • Pascal VOC12 segmentation dataset: Used for semantic segmentation.
  • MINC dataset: Used for material segmentation.
  • NYU depth dataset: Used for depth upsampling.
  • Berkeley Segmentation Dataset (BSDS500): Used for image denoising.
  • Assamese character dataset: Used for character recognition.
  • MNIST dataset: Used for demonstrating robustness to subsampled data.
  • 3D body meshes: Used for 3D mesh denoising.

Key Insights and Ideas

  • Beyond Gaussian: The most significant insight is that moving beyond fixed Gaussian kernels for bilateral filters unlocks substantial performance gains across various tasks. Learning arbitrary filter shapes allows for much greater adaptability to data characteristics.
  • Permutohedral Lattice as a General Convolution Engine: The permutohedral lattice, previously used for approximating Gaussian filters, proves to be a powerful and efficient framework for general high-dimensional sparse convolutions. Its ability to handle arbitrary feature spaces and sparse data is key.
  • End-to-End Learning of Image Adaptive Filters: By deriving gradients for the permutohedral convolution, the authors enable the direct learning of image-adaptive filters within deep learning frameworks. This bridges the gap between traditional filtering and modern neural network architectures.
  • BNNs as a Natural Extension: Integrating learned bilateral filters as layers in CNNs (BNNs) provides a principled way to incorporate structured, image-adaptive operations, offering advantages over standard fixed-receptive-field convolutions, especially for sparse or high-dimensional inputs.

Key Results

  • Improved Performance: Learned bilateral filters consistently outperform their Gaussian counterparts in various applications, including:
  • Image Upsampling: Achieved higher PSNR values for color and depth upsampling. For example, learned filters improved depth upsampling accuracy by 0.015 (p-value 0.015) over bicubic interpolation.
  • Semantic Segmentation: Achieved higher Intersection over Union (IoU) scores, with learned CRFs improving performance by 2.93% over Gaussian CRFs.
  • Material Segmentation: Showed improvements in pixel accuracy.
  • Image Denoising: Achieved higher PSNR values (e.g., 26.51 for learned bilateral vs. 26.27 for Gaussian).
  • 3D Mesh Denoising: Reduced vertex distance RMSE and normal angle error compared to Gaussian bilateral filtering.
  • Character Recognition: BNN variants of LeNet and DeepCNet achieved higher accuracy than their standard counterparts.
  • Faster Convergence: In tasks like character recognition, BNNs demonstrated faster convergence rates compared to standard CNNs with comparable parameter counts.
  • Efficiency on Sparse Data: The permutohedral lattice convolution, with its hashing mechanism, is shown to be more efficient (both in terms of runtime and memory) for sparse, high-dimensional data compared to standard spatial convolutions.

Limitations and Future Work

  • Computational Cost: While efficient for sparse data, the permutohedral lattice convolution can still be computationally intensive for dense, high-dimensional inputs, and matrix K can become too large for GPU memory, requiring sequential computation.
  • Data Requirements: For some tasks, like semantic segmentation, the performance gains from learned filters might be limited by the availability of training data, especially for infrequent classes.
  • Future Work: The authors suggest several avenues for future research:
  • Exploring more efficient implementations and optimizations for the permutohedral lattice convolution, especially for dense data.
  • Investigating the use of learned bilateral filters in more complex CNN architectures and for intermediate layers.
  • Equipping deconvolutional networks with bilateral filters for tasks like depth estimation.
  • Developing more sophisticated methods for handling multi-layer, "loose" mean-field inference in CRFs.
  • Exploring the potential of learned bilateral filters for representing more complex object structures beyond pixels, such as bounding boxes or segmentations.

Summary updated

Abstract

Bilateral filters have wide spread use due to their edge-preserving properties. The common use case is to manually choose a parametric filter type, usually a Gaussian filter. In this paper, we will generalize the parametrization and in particular derive a gradient descent algorithm so the filter parameters can be learned from data. This derivation allows to learn high dimensional linear filters that operate in sparsely populated feature spaces. We build on the permutohedral lattice construction for efficient filtering. The ability to learn more general forms of high-dimensional filters can be used in several diverse applications. First, we demonstrate the use in applications where single filter applications are desired for runtime reasons. Further, we show how this algorithm can be used to learn the pairwise potentials in densely connected conditional random fields and apply these to different image segmentation tasks. Finally, we introduce layers of bilateral filters in CNNs and propose bilateral neural networks for the use of high-dimensional sparse data. This view provides new ways to encode model structure into network architectures. A diverse set of experiments empirically validates the usage of general forms of filters.
BibTeX
@misc{kiefel2015sparse,
  author = {Martin Kiefel and Varun Jampani and Peter V. Gehler},
  title = {{Sparse Convolutional Networks using the Permutohedral Lattice}},
  year = {2015},
  eprint = {1503.04949},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1503.04949},
}
Figure