TL;DR
This paper introduces MPG, a novel conditional GAN that can generate photorealistic pizza images with fine-grained control over ingredients and viewing angles. By leveraging both real-world and synthetic data, MPG overcomes limitations of existing methods, enabling generation beyond the scope of natural training data.
What Problem Does This Paper Address?
Generating realistic images with precise control over multiple attributes is a significant challenge in computer vision. While Generative Adversarial Networks (GANs) have excelled at creating high-fidelity images of objects like faces and cars, they struggle with complex domains like food, where variations in ingredients, textures, and presentation are vast. Existing food image generation methods often rely on image-to-image translation or produce blurry results, lacking granular control over specific ingredients or visual attributes. A key hurdle is the scarcity of diverse, fully annotated real-world datasets that capture the intricate interplay of food content, geometric perspective, and visual style.
Methods
The authors propose the Multi-attribute Pizza Generator (MPG), an extension of the StyleGAN2 architecture. MPG introduces a Multi-Scale Multi-Attribute Encoder (MSMAE) that injects conditioning information about ingredients and view attributes into the generator at multiple scales. This allows for more nuanced control over the generated image. To address the lack of paired attribute labels in real-world datasets, MPG employs a cross-domain learning strategy. A geometric style attribute regressor is trained on a synthetic dataset (PizzaView) and then used to regularize the generator trained on a real-world dataset (Pizza10). This allows MPG to synthesize images with view attributes outside the range observed in the real training data. The framework also incorporates Attribute Regularizers (ARs) to ensure the generated images accurately reflect the specified ingredients and view.
Datasets
- Pizza10: A newly curated real-world pizza dataset with ingredient annotations. It was derived from the existing
pizzaGANdataby relabeling and removing ambiguous ingredients to improve data quality. - PizzaView: A novel CGI dataset created from 3D pizza models rendered in Blender. This dataset is specifically annotated with geometric style attributes (camera angle, scale, x-shift, y-shift) and is crucial for training the view attribute regressor.
Key Insights and Ideas
- Cross-domain learning for view synthesis: Training a view attribute regressor on synthetic data (PizzaView) and using it to regularize generation on real data (Pizza10) is an effective way to extend control beyond the natural range of real-world observations.
- Multi-scale attribute injection: Injecting attribute information at multiple scales within the generator (via MSMAE) leads to better control and more realistic synthesis.
- Disentanglement of attributes: The proposed method demonstrates a good degree of disentanglement between ingredients, view attributes, and visual style, allowing for independent control.
- Importance of data curation: The creation of Pizza10 highlights how careful curation of existing datasets can significantly improve performance on downstream tasks like ingredient classification.
Key Results
- MPG achieves significantly better performance (lower FID, higher mAP) compared to baseline text-to-image generation models like StackGAN2, CookGAN, and AttnGAN.
- The FID of MPG approaches the estimated lower bound for real images, indicating high image quality and diversity.
- MPG demonstrates strong control over ingredients, achieving an mAP of 0.9998 on its generated images.
- The model can generate images with view attributes outside the natural range of the Pizza10 dataset, as evidenced by low RMSE values for angle, scale, dx, and dy.
- Ablation studies show that both MSMAE and Attribute Regularizers are crucial for MPG's performance. Removing MSMAE increases FID and decreases mAP/RMSE, while removing ARs drastically reduces mAP/RMSE.
Limitations and Future Work
- Computational cost: The paper notes that training MPG is computationally expensive.
- Loss of fine detail: Some loss of detail at fine resolutions is observed, which the authors aim to investigate in future work.
- SeFa limitations: The paper mentions limitations of the SeFa method for attribute control, including the need for manual correlation of attributes with eigenvectors and the inability to find eigenvectors controlling ingredient appearance.
- Future work: The authors plan to investigate methods to reduce computational cost and improve fine-resolution detail. They also aim to explore more sophisticated attribute disentanglement techniques.
Summary updated