TL;DR
This paper introduces Spatial Layout Injection (SLI-pSp), a novel technique to enhance image-to-image translation by incorporating fine-grained spatial information into the StyleGAN generator. By injecting multi-scale encoder features into StyleGAN's noise layers, SLI-pSp significantly improves the generation of detailed features, outperforming existing methods like pSp, especially on complex datasets.
What Problem Does This Paper Address?
Generative Adversarial Networks (GANs), particularly StyleGAN, have revolutionized image generation. Image-to-image (I2I) translation aims to transform an image from a source domain to a target domain while preserving content and adapting style. While methods like pSp leverage pre-trained StyleGAN generators for I2I tasks, they often struggle with finer details (e.g., earrings, hats) and exhibit inconsistencies on complex datasets. This is attributed to their "global approach," which fails to adequately capture and propagate the spatial layout information present in the input image throughout the generation process. Existing methods that try to incorporate spatial information often do so by modifying the core style injection mechanism, potentially sacrificing global consistency or representational capacity.
Methods
The core contribution is the Spatial Layout Injection (SLI) technique, integrated into the pSp framework. Instead of altering pSp's map2style network (which downsamples spatial resolution to 1x1), SLI injects multi-scale feature maps from an encoder (specifically, a feature pyramid network built on a ResNet backbone) directly into the noise layers of the StyleGAN generator.
Here's a breakdown:
- Feature Extraction: A feature pyramid network extracts multi-scale feature maps from the input image.
- Spatial Layout Injection (SLI): These extracted feature maps are processed (e.g., via a convolution to match channel dimensions) and then concatenated with the existing noise layers at various spatial scales within the StyleGAN generator. This injection happens without modifying the
map2stylenetwork, preserving the global style propagation. - Attentive SLI (ASLI-pSp): A variant is proposed using Bottleneck Transformers to further enhance the spatial feature representation before injection, demonstrating the flexibility of the SLI concept.
The authors emphasize that this approach preserves the global aspects of image generation while enhancing local spatial details, a balance that was previously challenging.
Datasets
The paper evaluates SLI-pSp on several datasets across different I2I tasks:
- CelebAMask-HQ: Used for the "Segmentation Map to Face" task, this dataset provides face images with detailed segmentation masks.
- CelebA: Utilized for the "Super Resolution" task.
- AFHQ-Dog: Also used for "Super Resolution," this dataset features dog faces, offering a different domain than human faces.
- Places2-CustomBuildings: A custom dataset compiled from building categories within Places2, used for the "Edges to RGB Image" task. This dataset was chosen for its complexity and high variance in lighting, pose, and structure, serving as a stress test for the method.
Key Insights and Ideas
- The Importance of Spatial Layout: The paper strongly argues that for many I2I tasks, simply relying on global style information is insufficient. Capturing and injecting fine-grained spatial layout is crucial for generating realistic and detailed outputs.
- Balancing Global and Local Information: SLI-pSp successfully achieves a balance between preserving global image consistency (inherent in StyleGAN and pSp) and enhancing local, spatially-aware details. This is a significant improvement over methods that might overemphasize one at the expense of the other.
- Flexibility of SLI: The proposed SLI is presented as a concept rather than a rigid architecture. The authors demonstrate its adaptability by introducing an attention-based variant (ASLI-pSp), suggesting it can be modified based on computational constraints and desired performance.
- Noise Layers as Injection Points: Leveraging StyleGAN's noise layers for feature injection is a clever choice, as these layers are designed to introduce local stochastic variations, making them a natural place to inject spatially-aware information without disrupting the core style synthesis.
Key Results
SLI-pSp demonstrates significant improvements across various I2I tasks:
- Segmentation Map to Face: SLI-pSp and ASLI-pSp achieve substantially better FID scores (e.g., 37.32 and 36.89 respectively, compared to pSp's 53.90) and higher mean IoU scores (0.81 vs. 0.61 for pSp), indicating more realistic and semantically accurate face generation with better preservation of fine details like earrings and hairstyles.
- Edges to RGB Image: On the challenging Places2-CustomBuildings dataset, SLI-pSp drastically reduces FID from 236.733 (pSp) to 23.71, and ASLI-pSp to 28.71. Qualitatively, it captures intricate building details (window designs, engravings) that pSp misses.
- Super Resolution: SLI-pSp and ASLI-pSp show substantial improvements in FID scores (e.g., around 10.21 and 9.90 for 8x SR, compared to pSp's 31.35). They better preserve spatial information, hairstyles, and details like caps that pSp often misinterprets.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Sharpness Reduction in ASLI-pSp: For super-resolution tasks, the attention-based ASLI-pSp variant sometimes exhibits a slight reduction in sharpness and artifacts, possibly due to a strong locality bias.
- Ill-Posedness of Edge-to-Image: While SLI-pSp performs well on the edge-to-image task, the generated images still lack the finesse seen in facial images, suggesting room for improvement, especially given the inherent ill-posedness of inferring complex scenes from edges.
- Inversion for Future Research: The paper focuses on synthesis quality. Inversion (reconstructing an image from a latent code) is left as a future research direction.
- Potential for Discriminator: The authors speculate that incorporating a discriminator might further aid in achieving better visual quality, particularly for complex tasks.
- Exploring Architecture Modifications: Further probing of SLI block configurations and encoder modifications is suggested to improve performance.
Summary updated