Towards Reliable Image Outpainting: Learning Structure-Aware Multimodal Fusion with Depth Guidance

Lei Zhang, Chunyu Lin, Kang Liao, Yao Zhao

TL;DR

Researchers from Beijing Jiaotong University have developed a new framework for "reliable" image outpainting that uses sparse LiDAR depth data to guide the generation of missing pixels. By moving beyond mere visual plausibility to structural authenticity, their Depth-Guided Outpainting Network (DGONet) significantly improves both image quality and the performance of downstream tasks like 2D and 3D object detection.

What Problem Does This Paper Address?

Image outpainting—the task of extrapolating an image beyond its original boundaries—has traditionally focused on creating "visually plausible" results. While these results look good to the human eye, they are often fictitious hallucinations that don't reflect the actual scene.

In fields like autonomous driving, this lack of authenticity is a dealbreaker. We don't just need a "plausible" extension of a street; we need to know if there is actually a car or a building in the blind spot. The authors address this by leveraging the fact that LiDAR sensors typically have a much wider field of view (FOV) than cameras. By using the sparse but authentic structural data from LiDAR, they aim to guide the outpainting process toward reality rather than fiction.

A comparison of image outpainting tasks: (a) general image outpainting and (b) the proposed depth-guided image outpainting.

Methods

The core of the proposal is the Depth-Guided Outpainting Network (DGONet), which consists of two primary innovations:

  1. Multimodal Learning Module (ML): This module handles the distinct characteristics of RGB and depth data.
  2. Depth Branch: LiDAR data is notoriously sparse (often only ~7% valid pixels). Standard convolutions treat all pixels equally, which "dilutes" the signal with zeros. The authors use Partial Convolutions (PConv) and a sparse depth mask to ensure the network only learns from valid data points.
  3. RGB Branch: This uses Gated Convolutions, which allow the network to dynamically select features at each spatial location, making it better at "filling holes" in the incomplete image.
  4. Depth Guidance Fusion Module (DGF): Instead of simply concatenating RGB and depth features (a common but "weak" interaction), the DGF uses a progressive fusion scheme. It treats depth features as dynamic filtering kernels that are applied to the RGB features. This ensures the generated visual content is strictly aligned with the underlying 3D structure.

To train the model, the authors introduced a specialized Cross-modal Loss, which forces the high-level semantics of the extrapolated RGB regions to match the semantics of the depth modality, and an Edge Loss to ensure sharp object boundaries.

An overview of the network architecture showing the dual-branch encoder and the fusion process.

Datasets

The model was trained and evaluated on two major autonomous driving benchmarks:

  • KITTI Dataset: A standard for mobile robotics and autonomous driving, providing synchronized video and LiDAR point clouds.
  • Waymo Open Dataset: A large-scale autonomous driving dataset with high-quality sensor data.

The researchers expanded images from a central $256 \times 256$ patch to a full $256 \times 512$ scene, using the LiDAR data available for the expanded regions as guidance.

Key Insights and Ideas

  • Authenticity over Plausibility: The shift from "hallucinating" pixels to "reconstructing" them based on physical sensor data is a significant step toward making generative models useful in safety-critical applications.
  • Handling Sparsity is Key: The visualization in Figure 3 shows that without the specialized Multimodal Learning (ML) module, depth features remain noisy and scattered. With ML, the network successfully extracts coherent structural outlines from sparse points.
  • Generative Models as Data Augmentation: By producing reliable outpainted images, this method can be used to enhance datasets for other tasks, like training object detectors to see further or wider than the original camera hardware allowed.
Feature visualization of the Depth-domain Branch, showing how the ML module clarifies structural outlines.

Key Results

DGONet outperformed state-of-the-art (SOTA) methods like OPDA and Boundless across multiple metrics:

  • Perceptual Quality: Achieved the best scores in FID (Fréchet Inception Distance) and LPIPS, indicating the results are more consistent with human perception. On KITTI, DGONet reached an FID of 25.52 compared to OPDA's 26.80.
  • Reliability (Object Detection): This is the most telling result. The authors tested how well a pre-trained vehicle detector performed on the generated parts of the image. DGONet achieved a 79.23% AP (Average Precision) on KITTI, significantly higher than the 74.75% achieved by the previous best method.
  • 3D Consistency: The method also showed improvements in 3D object detection metrics (AP3D), proving that the generated RGB content is spatially accurate enough to support 3D perception.

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Image outpainting technology generates visually plausible content regardless of authenticity, making it unreliable to be applied in practice. Thus, we propose a reliable image outpainting task, introducing the sparse depth from LiDARs (Light Detection And Ranging devices) to extrapolate authentic RGB scenes. The large field view of LiDARs allows it to serve for data enhancement and further multimodal tasks. Concretely, we propose a Depth-Guided Outpainting Network to model different feature representations of two modalities and learn the structure-aware cross-modal fusion. And two components are designed: 1) The Multimodal Learning Module produces unique depth and RGB feature representations from the perspectives of different modal characteristics. 2) The Depth Guidance Fusion Module leverages the complete depth modality to guide the establishment of RGB contents by progressive multimodal feature fusion. Furthermore, we specially design an additional constraint strategy consisting of Cross-modal Loss and Edge Loss to enhance ambiguous contours and expedite reliable content generation. Extensive experiments on KITTI and Waymo datasets demonstrate our superiority over the state-of-the-art method, quantitatively and qualitatively.
BibTeX
@inproceedings{zhang2023towards,
  author = {Lei Zhang and Chunyu Lin and Kang Liao and Yao Zhao},
  title = {{Towards Reliable Image Outpainting: Learning Structure-Aware Multimodal Fusion with Depth Guidance}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095104},
}
Figure