Shadow-Enlightened Image Outpainting

Hang Yu, Ruilin Li, Shaorong Xie, Jiayan QiuCVPR 2024 · DOI

Introduction

Image outpainting is the process of extending the visual content of an image beyond its original boundaries. While significant progress has been made in maintaining textural and semantic consistency, current models often struggle with physical plausibility. This paper introduces a novel approach that leverages shadows as "hidden information" to infer the presence and characteristics of objects located outside the visible frame.

Problem Statement: The Missing Physical Cues

The primary difficulty in outpainting lies in hallucinating content that is entirely absent from the input. Most existing methods rely on local context—such as extending a patch of grass or a sky gradient. However, they frequently overlook shadows cast by out-of-frame objects. In real-world scenes, these shadows provide critical geometric cues about the shape, category, and position of invisible entities. Ignoring these cues leads to "misaligned" results where generated objects do not correspond to the existing lighting and shadow patterns of the scene.

Methodology: The Shadow-Enlightened Pipeline

The authors propose a three-stage pipeline designed to extract, refine, and utilize shadow information for more accurate scene expansion.

Overview of shadow alignment and extraction challenges.

1. Shadow Extraction and Refinement

Using pretrained detectors, the model identifies shadow regions. Because real-world shadows are often fragmented or overlapping, the authors developed specific algorithms for Shadow Merging (to reconnect shadows split by occluders) and Shadow Splitting (to distinguish between shadows from different sources). Additionally, a "purification" step removes underlying textures from the shadow regions to isolate their geometric properties.

2. Outpainting with Shadows

The refined shadow features are integrated into a Scene Layout Expansion module. By analyzing the shadow's geometry alongside the existing scene graph, the model predicts the most probable layout of invisible objects. These layouts are then synthesized into pixels using a layout-to-image diffusion model.

3. Alignment Discriminator

To ensure physical consistency, the pipeline employs local and global discriminators. These components verify that the newly generated objects align correctly with the global light source and the specific shadows that prompted their creation, preventing visual artifacts.

Key Insights

  • Shadows as Geometric Projections: Rather than treating shadows as noise, this method treats them as projections of unobserved data, transforming outpainting from a generative "guessing game" into a constrained reconstruction task.
  • Illumination Awareness: By incorporating shadow masks, the model inherently learns the global light direction, ensuring that the lighting of the extended region matches the original image.
  • Iterative Optimization: The "Merging and Splitting" algorithm is crucial for converting noisy raw detections into usable instance-level features.

Experimental Results

The proposed method, SGT+, was evaluated against the state-of-the-art (SOTA) SGT model on the Visual Genome and COCO-stuff benchmarks.

  • Layout Accuracy: On Visual Genome, the Mean Intersection over Union (mIoU) improved from 22.7 to 28.4.
  • Image Realism: The Fréchet Inception Distance (FID) decreased from 20.12 to 19.47, indicating higher visual quality.
  • Human Evaluation: In user studies, 87% of participants preferred the shadow-aware results, citing better authenticity and alignment.

Conclusion and Future Work

By treating shadows as a source of enlightenment rather than a nuisance, this research significantly improves the semantic and physical consistency of image outpainting. Future work may explore applying these techniques to dynamic video environments or utilizing shadow cues for predictive tasks like pedestrian trajectory forecasting.

Summary updated

BibTeX
@inproceedings{yu2024shadow,
  author = {Hang Yu and Ruilin Li and Shaorong Xie and Jiayan Qiu},
  title = {{Shadow-Enlightened Image Outpainting}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00750},
}
Figure