Abstract
Traditional image outpainting focuses on generating visually plausible textures, often resulting in "hallucinated" content that lacks structural authenticity. This paper introduces DGONet, a framework designed for reliable scene reconstruction by leveraging sparse LiDAR depth data. By integrating a Multimodal Learning module and a Depth Guidance Fusion module, DGONet ensures that extrapolated peripheral content is both visually realistic and structurally accurate. Evaluations on the KITTI and Waymo datasets demonstrate significant improvements in downstream 2D and 3D object detection tasks, proving the utility of depth-guided outpainting for safety-critical applications like autonomous driving.
Introduction: From Hallucination to Reliability
Image outpainting—the process of extrapolating an image beyond its original boundaries—has traditionally been a "fictitious" task. Standard generative models, such as GANs, excel at creating textures that appear correct to the human eye but often bear no relation to the actual reality of the scene. While sufficient for digital art, this lack of authenticity limits the utility of outpainting in safety-critical domains like autonomous driving.
The authors identify a unique opportunity: LiDAR sensors typically possess a much broader field of view (FOV) than cameras. However, LiDAR data is inherently sparse (often only ~7% density) and represents depth rather than RGB intensity. The challenge lies in effectively fusing these disparate modalities to "guide" the camera's outpainting so that generated pixels accurately represent the physical environment.
The DGONet Framework
The proposed Depth-Guided Outpainting Network (DGONet) introduces two primary architectural innovations to bridge the gap between sparse depth and dense RGB data:
- Multimodal Learning Module (ML): To handle the distinct characteristics of RGB and LiDAR, the network employs two specialized branches. The Depth-domain branch utilizes Partial Convolutions to prevent invalid (empty) LiDAR pixels from polluting the feature extraction process. Simultaneously, the RGB-domain branch uses Gated Convolutions, allowing the network to learn a dynamic selection mechanism for filling in the missing peripheral regions.
- Depth Guidance Fusion Module (DGF): Rather than simple concatenation, the DGF uses depth features to generate dynamic filtering kernels. These kernels are applied to the RGB features, ensuring that the structural "skeleton" provided by the LiDAR directly dictates the establishment of RGB content.
To maintain sharpness and semantic accuracy, the framework incorporates a Cross-modal Loss, which aligns high-level generated features with the depth modality, and an Edge Loss using Canny filters to preserve the sharp outlines of objects like vehicles and infrastructure.
Experimental Results
DGONet was evaluated on the KITTI and Waymo Open Datasets, expanding 256×256 images into 256×512 panoramas. The model significantly outperformed state-of-the-art (SOTA) methods like SpiralNet and OPDA across both visual and functional metrics:
- Visual Quality: DGONet achieved the lowest FID (25.52 on KITTI) and LPIPS scores, indicating superior alignment with real-world image distributions.
- Reliability and Downstream Performance: In 2D vehicle detection, DGONet reached an Average Precision (AP) of 79.23% on KITTI, a substantial improvement over baselines (66-74%). The model also showed superior performance in 3D object detection, confirming that the generated RGB content is spatially consistent with the underlying 3D structure.
Conclusion and Future Work
DGONet shifts the paradigm of image outpainting from artistic hallucination to reliable reconstruction. By treating sparse depth as a structural scaffold rather than just another input channel, the model produces functional data suitable for autonomous perception. Future research may address the system's dependency on active LiDAR data and the computational overhead associated with dynamic kernel generation to enable real-time deployment on edge devices.
Summary updated