Multi-attribute Pizza Generator: Cross-domain Attribute Control with Conditional StyleGAN

Fangda Han, Guoyao Hao, Ricardo Guerrero, Vladimir Pavlovic

TL;DR

This paper introduces MPG, a novel conditional GAN that can generate photorealistic pizza images with fine-grained control over ingredients and viewing angles. By leveraging both real-world and synthetic data, MPG overcomes limitations of existing methods, enabling generation beyond the scope of natural training data.

What Problem Does This Paper Address?

Generating realistic images with precise control over multiple attributes is a significant challenge in computer vision. While Generative Adversarial Networks (GANs) have excelled at creating high-fidelity images of objects like faces and cars, they struggle with complex domains like food, where variations in ingredients, textures, and presentation are vast. Existing food image generation methods often rely on image-to-image translation or produce blurry results, lacking granular control over specific ingredients or visual attributes. A key hurdle is the scarcity of diverse, fully annotated real-world datasets that capture the intricate interplay of food content, geometric perspective, and visual style.

Methods

The authors propose the Multi-attribute Pizza Generator (MPG), an extension of the StyleGAN2 architecture. MPG introduces a Multi-Scale Multi-Attribute Encoder (MSMAE) that injects conditioning information about ingredients and view attributes into the generator at multiple scales. This allows for more nuanced control over the generated image. To address the lack of paired attribute labels in real-world datasets, MPG employs a cross-domain learning strategy. A geometric style attribute regressor is trained on a synthetic dataset (PizzaView) and then used to regularize the generator trained on a real-world dataset (Pizza10). This allows MPG to synthesize images with view attributes outside the range observed in the real training data. The framework also incorporates Attribute Regularizers (ARs) to ensure the generated images accurately reflect the specified ingredients and view.

Datasets

  • Pizza10: A newly curated real-world pizza dataset with ingredient annotations. It was derived from the existing pizzaGANdata by relabeling and removing ambiguous ingredients to improve data quality.
  • PizzaView: A novel CGI dataset created from 3D pizza models rendered in Blender. This dataset is specifically annotated with geometric style attributes (camera angle, scale, x-shift, y-shift) and is crucial for training the view attribute regressor.

Key Insights and Ideas

  • Cross-domain learning for view synthesis: Training a view attribute regressor on synthetic data (PizzaView) and using it to regularize generation on real data (Pizza10) is an effective way to extend control beyond the natural range of real-world observations.
  • Multi-scale attribute injection: Injecting attribute information at multiple scales within the generator (via MSMAE) leads to better control and more realistic synthesis.
  • Disentanglement of attributes: The proposed method demonstrates a good degree of disentanglement between ingredients, view attributes, and visual style, allowing for independent control.
  • Importance of data curation: The creation of Pizza10 highlights how careful curation of existing datasets can significantly improve performance on downstream tasks like ingredient classification.

Key Results

  • MPG achieves significantly better performance (lower FID, higher mAP) compared to baseline text-to-image generation models like StackGAN2, CookGAN, and AttnGAN.
  • The FID of MPG approaches the estimated lower bound for real images, indicating high image quality and diversity.
  • MPG demonstrates strong control over ingredients, achieving an mAP of 0.9998 on its generated images.
  • The model can generate images with view attributes outside the natural range of the Pizza10 dataset, as evidenced by low RMSE values for angle, scale, dx, and dy.
  • Ablation studies show that both MSMAE and Attribute Regularizers are crucial for MPG's performance. Removing MSMAE increases FID and decreases mAP/RMSE, while removing ARs drastically reduces mAP/RMSE.

Limitations and Future Work

  • Computational cost: The paper notes that training MPG is computationally expensive.
  • Loss of fine detail: Some loss of detail at fine resolutions is observed, which the authors aim to investigate in future work.
  • SeFa limitations: The paper mentions limitations of the SeFa method for attribute control, including the need for manual correlation of attributes with eigenvectors and the inability to find eigenvectors controlling ingredient appearance.
  • Future work: The authors plan to investigate methods to reduce computational cost and improve fine-resolution detail. They also aim to explore more sophisticated attribute disentanglement techniques.

Summary updated

Abstract

Multi-attribute conditional image generation is a challenging problem in computervision. We propose Multi-attribute Pizza Generator (MPG), a conditional Generative Neural Network (GAN) framework for synthesizing images from a trichotomy of attributes: content, view-geometry, and implicit visual style. We design MPG by extending the state-of-the-art StyleGAN2, using a new conditioning technique that guides the intermediate feature maps to learn multi-scale multi-attribute entangled representationsof controlling attributes. Because of the complex nature of the multi-attribute image generation problem, we regularize the image generation by predicting the explicit conditioning attributes (ingredients and view). To synthesize a pizza image with view attributesoutside the range of natural training images, we design a CGI pizza dataset PizzaView using 3D pizza models and employ it to train a view attribute regressor to regularize the generation process, bridging the real and CGI training datasets. To verify the efficacy of MPG, we test it on Pizza10, a carefully annotated multi-ingredient pizza image dataset. MPG can successfully generate photo-realistic pizza images with desired ingredients and view attributes, beyond the range of those observed in real-world training data.
BibTeX
@inproceedings{han2021multi,
  author = {Fangda Han and Guoyao Hao and Ricardo Guerrero and Vladimir Pavlovic},
  title = {{Multi-attribute Pizza Generator: Cross-domain Attribute Control with Conditional StyleGAN}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=KHKYG4BbP33},
}
Figure