Multimodal Image Outpainting With Regularized Normalized Diversification

Lingzhi Zhang, Jiancong Wang, Jianbo ShiWACV 2020

TL;DR

This paper tackles the challenge of "image outpainting," where the goal is to generate diverse and realistic backgrounds for a given foreground object. They introduce a novel diversity regularization technique to combat mode collapse in generative models and a feature pyramid discriminator to enhance image quality, achieving state-of-the-art results in generating varied and high-fidelity outpainted images.

What Problem Does This Paper Address?

Imagine you're shopping for a couch online. You can easily picture it in various settings – a cozy living room, a modern apartment, an office. This ability to imagine different contexts for an object is what humans do effortlessly. This paper addresses the computational challenge of enabling machines to do the same, a task they call "image outpainting." Specifically, they focus on generating a set of diverse and plausible backgrounds when given only a small foreground object. Traditional generative models, like GANs, often suffer from "mode collapse," meaning they tend to produce very similar outputs even when given different inputs or when trying to generate variety. While some methods try to maintain diversity by preserving distances between generated samples, they don't explicitly prevent outputs from different conditional inputs from collapsing into a few common modes. This paper aims to solve this by encouraging not just plausible but also diverse outputs for image outpainting.

Methods

The core of their approach lies in two main innovations:

  • Regularized Normalized Diversification: Building upon existing "normalized diversification" techniques that preserve pairwise distances between generated samples, they introduce a new regularization loss. This loss aims to pull the sampled outputs of different conditional inputs away from each other, directly addressing the issue of mode collapse across distinct inputs. This is achieved by enforcing a hard constraint using pixel-wise Euclidean distance between generated outputs and ground truth images, effectively encouraging diversity across different conditional inputs.
  • Feature Pyramid Discriminator (FPD): To improve the visual quality of the generated images, they propose a novel discriminator architecture. Inspired by the Pyramid Scene Parsing Network (PSPNet), the FPD explicitly computes features at multiple scales within the discriminator. This allows it to check both low-level texture realism and high-level structural semantics, providing a more robust supervisory signal to the generator.

The overall optimization objective combines the adversarial loss (for realism), the diversity loss (for diversity within a single input), and their proposed diversity regularization loss (for diversity across different inputs), along with the FPD.

Datasets

The authors evaluate their method on two real-world datasets:

  • CelebA: A dataset of celebrity faces. For this task, they extract facial landmarks (eyes and nose) as the foreground input.
  • Cityscapes: A dataset of urban street scenes. For this task, they use instance segmentation to extract foreground objects (like cars or pedestrians) as inputs.

They also conducted experiments on a synthetic dataset designed to model a non-linear mapping from a uniform input space to a star-shaped output space, specifically to test the diversity generation capabilities.

Key Insights and Ideas

  • Inverse Affordance: The paper frames image outpainting as "inverse affordance learning." Instead of learning how objects interact with their environment (affordance learning), they are learning to hallucinate the environment for a given object.
  • Diversity is Crucial for Outpainting: Unlike traditional inpainting where much of the scene is given, outpainting requires filling large missing regions, demanding a higher degree of semantic understanding and thus, greater output diversity.
  • Mode Collapse is a Cross-Input Problem: The key insight is that existing diversity methods often focus on diversity within a single input's outputs. Their work highlights the need to ensure diversity across outputs generated from different conditional inputs, which is critical for tasks like image outpainting.
  • Multi-Scale Discrimination for Quality: The FPD demonstrates that explicitly considering features at different resolutions within the discriminator leads to improved image realism and structural coherence.

Key Results

  • Synthetic Data: Quantitatively, their model achieved the highest "Number of Modes" and a competitive "Pairwise Distance" on synthetic data, indicating superior diversity compared to baselines like BicycleGAN, VAE-GAN, MSGAN, and NDiv. While VAE-GAN showed slightly better plausibility (Frechet Distance), it generated far fewer modes.
  • Real Image Data (CelebA & Cityscapes):
    • Diversity: Their method consistently achieved the highest diversity scores (measured by LPIPS) on both datasets, outperforming all baselines.
  • Quality: They achieved comparable or better image quality (measured by FID) than most baselines, with their full model (including FPD) achieving the best FID scores on both datasets.
  • Qualitative Results: Visualizations (Figures 7, 8, 9, 10) clearly show their model generating a wider range of plausible backgrounds and variations (e.g., diverse hairstyles and facial expressions in CelebA, varied street scenes in Cityscapes) compared to other methods, which often exhibit mode collapse.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method significantly improves diversity, some minor mode collapse might still occur. The quantitative evaluation on synthetic data shows VAE-GAN having a slightly better Frechet Distance, suggesting a potential trade-off between extreme diversity and absolute realism in some cases.
  • Future Work: The authors suggest potential applications in occluded face recognition for forensic purposes or general image editing. They also propose extending their work to understand object affordance within detection and segmentation tasks, indicating a broader impact for their research in computer vision.

Summary updated

BibTeX
@inproceedings{zhang2020multimodal,
  author = {Lingzhi Zhang and Jiancong Wang and Jianbo Shi},
  title = {{Multimodal Image Outpainting With Regularized Normalized Diversification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure