SSGP: Sparse Spatial Guided Propagation for Robust and Generic Interpolation

René Schuster, Oliver Wasenmüller, Christian Unger, Didier StrickerWACV 2020 · OpenReview

Abstract

Interpolation of sparse pixel information towards a dense target resolution finds its application across multiple disciplines in computer vision. State-of-the-art interpolation of motion fields applies model-based interpolation that makes use of edge information extracted from the target image. For depth completion, data-driven learning approaches are widespread. Our work is inspired by latest trends in depth completion that tackle the problem of dense guidance for sparse information. We extend these ideas and create a generic cross-domain architecture that can be applied for a multitude of interpolation problems like optical flow, scene flow, or depth completion. In our experiments, we show that our proposed concept of Sparse Spatial Guided Propagation (SSGP) achieves improvements to robustness, accuracy, or speed compared to specialized algorithms.

Introduction

In computer vision, many sensing and processing modalities yield "sparse" or "holey" data. For instance, LiDAR sensors provide depth measurements for only a small fraction of image pixels, and feature-matching algorithms often produce motion vectors for only a few distinct landmarks. To utilize this data effectively for tasks like autonomous driving or robotic navigation, these sparse points must be interpolated into a dense, high-resolution grid.

Overview of the interpolation task: (a) Input RGB image, (b) Sparse LiDAR measurements, and (c) the resulting dense depth map produced by SSGP.

Historically, interpolation has been addressed through fragmented, domain-specific methods. Optical flow research often relied on hand-crafted, edge-aware models, while depth completion favored data-driven deep networks. Furthermore, many existing methods lack robustness, struggling when input density drops significantly or when data is corrupted by noise. SSGP (Sparse Spatial Guided Propagation) bridges this gap with a unified architecture designed for robust, multi-domain interpolation.

Methodology

The core of SSGP is a generic sparse-to-dense encoder-decoder architecture, typically a U-Net variant, enhanced by several key innovations:

Dual-Codec Architecture

The network employs two parallel paths: an RGB Codec and a Sparse Data Codec. The RGB Codec extracts structural features from the high-resolution reference image, which are then used to guide the interpolation process in the Sparse Data Codec.

The SSGP architecture consists of an RGB codec (top) that provides guidance to the sparse data codec (bottom) via affinity blocks at multiple scales.

Sparsity-Aware Convolution

Standard convolutions often smudge valid data by treating "empty" pixels as zeros. SSGP utilizes sparsity-aware convolutions that track valid pixels via a binary mask. This ensures that the convolution output is normalized based on the local density of valid information, preventing artifacts in sparse regions.

Image-Guided Propagation and Flat Affinities

The network predicts spatially variant kernels, or affinity maps, based on the RGB features. These kernels dictate how information should spread—for example, preventing depth or motion values from "bleeding" across object boundaries. To maintain efficiency and prevent overfitting, SSGP uses "flat" affinity maps (one per scale) rather than per-channel kernels, significantly reducing parameter count without sacrificing performance.

Dense Refinement

Following the initial interpolation, the model performs several iterations of spatial propagation to refine the output. This step ensures local consistency and sharpens edges in the final dense map.

Experimental Evaluation

The model was rigorously tested across several industry-standard benchmarks:

  • KITTI (2015 & Depth Completion): Real-world outdoor driving data for depth, optical flow, and scene flow.
  • FlyingThings3D (FT3D): A large-scale synthetic dataset used for pre-training.
  • Sintel & HD1K: Challenging datasets for evaluating generalization in optical flow.

Key Results

  • Cross-Domain Performance: SSGP demonstrates that interpolation for 2D motion (flow) and 3D geometry (depth) can be unified. On the KITTI benchmark, it reduced End-Point Error (EPE) in optical flow by approximately 50% compared to previous interpolators like InterpoNet.
  • Robustness to Sparsity and Noise: The model is remarkably stable. In scene flow tests, error rates remained nearly constant even as input density was reduced from 100% to just 1%. It also showed superior resistance to Gaussian and Laplacian noise compared to traditional heuristic methods.
  • Computational Efficiency: By replacing hand-crafted heuristics with a trainable deep model, SSGP achieves a 5x to 20x speedup over competitors like EPICFlow, processing frames in approximately 0.19 seconds.

Conclusion

SSGP provides a robust and generic framework for sparse data interpolation. While it excels in real-world scenarios and high-density data, performance on synthetic datasets like Sintel is currently on par with existing methods, likely due to limited scene variance. Future work will focus on optimizing the refinement stage and exploring one-pass strategies to further reduce computational overhead while maintaining high interpolation quality.

Summary updated

Abstract

Interpolation of sparse pixel information towards a dense target resolution finds its application across multiple disciplines in computer vision. State-of-the-art interpolation of motion fields applies model-based interpolation that makes use of edge information extracted from the target image. For depth completion, data-driven learning approaches are widespread. Our work is inspired by latest trends in depth completion that tackle the problem of dense guidance for sparse information. We extend these ideas and create a generic cross-domain architecture that can be applied for a multitude of interpolation problems like optical flow, scene flow, or depth completion. In our experiments, we show that our proposed concept of Sparse Spatial Guided Propagation (SSGP) achieves improvements to robustness, accuracy, or speed compared to specialized algorithms.
BibTeX
@inproceedings{schuster2020ssgp,
  author = {René Schuster and Oliver Wasenmüller and Christian Unger and Didier Stricker},
  title = {{SSGP: Sparse Spatial Guided Propagation for Robust and Generic Interpolation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=TheSMKAFPc7},
}
Figure