Depth-SIMS: Semi-Parametric Image and Depth Synthesis

Valentina Musat, Daniele De Martini, Matthew Gadd, Paul Newman

TL;DR

Depth-SIMS is a semi-parametric image synthesis framework that generates high-quality RGB images alongside perfectly aligned dense depth maps. By combining real-world "object blobs" with a neural in-painting network, it produces synthetic training data that significantly improves the performance of downstream tasks like semantic segmentation and depth completion.

What Problem Does This Paper Address?

Training deep learning models for autonomous driving requires massive amounts of pixel-perfect ground truth (GT) data, which is notoriously expensive and slow to annotate manually. While simulators (like CARLA) or pure generative models (GANs) can create data, they often suffer from a "reality gap" or structural inconsistencies.

Previous "semi-parametric" methods like SIMS tried to bridge this gap by stitching together real image pieces (blobs) onto a canvas. However, these methods often struggled with two things:

  1. Alignment: The final synthesized image didn't always match the underlying segmentation mask perfectly.
  2. Modality: They focused almost exclusively on RGB, ignoring the crucial depth information needed for self-driving stacks.

Methods

The authors propose a four-stage pipeline to generate "RGB-D" (Red, Green, Blue, and Depth) data:

  1. Object Blob Database: They decompose the Cityscapes dataset into a library of "blobs" (cars, pedestrians, etc.). To make searching this library fast, they use Hu moments—mathematical descriptors of shape—allowing them to quickly find a real object that fits a target shape.
  2. Canvasing: Using a "guiding" layout, they paste these blobs onto a blank canvas. They implement a smart hole-filling strategy for the segmentation map to ensure no "empty" pixels remain.
  3. Sparse Depth Composition: They don't just paste color; they also sample depth values from the original source data to create a sparse depth map that is perfectly aligned with the new RGB canvas.
  4. In-painting & Synthesis: They use a GAN-based in-painter (based on the SPADE architecture) and a segmenting discriminator (OASIS). This network does two things at once: it blends the edges of the pasted blobs to make the image look natural and "densifies" the sparse depth map into a complete, pixel-wise depth image.

Datasets

  • Cityscapes: Used as the primary source for object blobs, guiding layouts, and initial training.
  • A2D2 (Audi Autonomous Driving Dataset): Used as an "out-of-domain" test set to see how well models trained on Depth-SIMS data generalize to new environments.
  • KITTI: Used to evaluate the quality of the synthesized depth maps for the depth completion task.

Key Insights and Ideas

  • Alignment > Perceptual Quality: A major finding is that for training other models, it is more important that the image perfectly matches the label (alignment) than it is for the image to look 100% "real" to a human (FID score). Depth-SIMS prioritizes this structural alignment.
  • Hu Moments for Speed: Replacing the standard pair-wise IoU (Intersection over Union) matching with Hu moments significantly speeds up the retrieval of object blobs from large databases without sacrificing shape accuracy.
  • Multi-modal In-painting: By forcing the network to predict both RGB and Depth simultaneously, the model learns a more robust representation of the scene's geometry, which actually helps the image quality.

Key Results

  • Better Alignment: Depth-SIMS achieved a 3.7 percentage point increase in mIoU (mean Intersection over Union) over previous state-of-the-art methods, indicating much tighter alignment between images and their labels.
  • Competitive Realism: It maintained a highly competitive FID score of 46.3, outperforming many parametric GANs in terms of visual quality.
  • Downstream Success: Models trained on Depth-SIMS data performed better on the A2D2 dataset than those trained on data from SIMS or OASIS.
  • Depth Accuracy: In depth completion tasks, the synthesized depth maps achieved a Root Mean Squared Error (RMSE) of 3.90m, significantly better than using raw "guiding" depth (5.99m) or other synthetic baselines.

Limitations and Future Work

  • Heuristic Ordering: The current system uses a simple ordering (e.g., "paste sky first, then buildings, then cars") to handle object occlusion. The authors suggest a more learned approach to object ordering could improve complex scenes.
  • Static vs. Dynamic: The method assumes static classes (road, vegetation) are texture-based and easy to in-paint, while dynamic classes (cars) must be pasted. This distinction is manual and could be made more flexible.
  • Source Dependency: The quality of the output is still bounded by the variety and quality of the "blobs" available in the source database. Expanding this to more diverse datasets is a logical next step.

Summary updated

Abstract

In this paper we present a compositing image synthesis method that generates RGB canvases with well aligned segmentation maps and sparse depth maps, coupled with an in-painting network that transforms the RGB canvases into high quality RGB images and the sparse depth maps into pixel-wise dense depth maps. We benchmark our method in terms of structural alignment and image quality, showing an increase in mIoU over SOTA by 3.7 percentage points and a highly competitive FID. Furthermore, we analyse the quality of the generated data as training data for semantic segmentation and depth completion, and show that our approach is more suited for this purpose than other methods.
BibTeX
@inproceedings{musat2022depth,
  author = {Valentina Musat and Daniele De Martini and Matthew Gadd and Paul Newman},
  title = {{Depth-SIMS: Semi-Parametric Image and Depth Synthesis}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=iUGVt7BJ2l},
}

Loading related content...

Figure