CookGAN: Meal Image Synthesis from Ingredients

Fangda Han, Ricardo Guerrero, Vladimir Pavlovic

TL;DR

This paper introduces CookGAN, a novel framework for generating realistic meal images solely from a list of ingredients. By combining an attention-based association model with a conditional GAN and incorporating a cycle-consistency constraint, CookGAN effectively tackles the complexity of food imagery, producing visually appealing and ingredient-accurate meal representations.

What Problem Does This Paper Address?

Generating realistic images from textual descriptions is a well-studied problem, but it becomes significantly more challenging when dealing with complex subjects like meals. Unlike simpler objects (e.g., birds, flowers), meal images are characterized by a high degree of variability due to factors like cooking methods, preparation styles, and the interplay of multiple ingredients. Existing text-to-image synthesis models often struggle to capture this complexity, leading to unrealistic or inaccurate food images. This paper aims to bridge this gap by developing a model that can synthesize photo-realistic meal images that accurately reflect a given list of ingredients.

Methods

CookGAN employs a two-stage approach:

  1. Attention-based Association Model: This component learns a shared latent space (dubbed "FoodSpace") where ingredient lists and meal images can be represented and compared. It uses a bidirectional LSTM with an attention mechanism to weigh the importance of individual ingredients, recognizing that not all ingredients contribute equally to visual appearance. This model is trained to associate correct ingredient-image pairs while penalizing incorrect ones.
  2. Generative Meal Image Network (CookGAN): This is a conditional Generative Adversarial Network (GAN), specifically inspired by StackGAN-v2, which generates images at multiple resolutions. The ingredient features extracted by the association model are used to condition the GAN. A key addition is a cycle-consistency constraint. This constraint ensures that a generated meal image, when passed back through the image encoder of the association model, results in a feature representation that is similar to the feature representation of the original ingredient list. This helps to enforce that the generated image truly "contains" the specified ingredients.

Datasets

The primary dataset used is Recipe1M [22], which contains approximately 1 million recipes with associated titles, instructions, ingredients, and images. For their experiments, the authors filtered this dataset to include recipes with at least one image and a reasonable number of ingredients/instructions. They also processed the ~16,000 unique ingredients in Recipe1M to create a more manageable list of 1989 "canonical ingredients" by merging similar terms and using word embeddings. For the meal image generation experiments, they focused on specific subsets of Recipe1M: salad, cookie, and muffin categories, due to their relatively consistent cooking methods and recognizable key ingredients.

Key Insights and Ideas

  • Ingredient Importance Varies: The attention mechanism is crucial because not all ingredients are equally visually prominent. Some ingredients (like spices) might be less visible than others (like a main protein or vegetable).
  • FoodSpace is Key: Learning a joint latent space (FoodSpace) that bridges the gap between textual ingredient descriptions and visual meal representations is fundamental.
  • Cycle-Consistency for Realism: The cycle-consistency constraint is vital for improving image quality and ensuring that the generated images are not just visually plausible but also semantically aligned with the input ingredients. It acts as a strong regularizer.
  • Canonical Ingredients Matter: Pre-processing ingredients into a canonical vocabulary significantly improves the model's ability to condition the image generation process, leading to better text-to-image alignment.

Key Results

  • Improved Retrieval: The attention-based association model, especially when using canonical ingredients, shows improved performance in retrieving corresponding images for ingredient queries (and vice-versa) compared to baseline methods, as measured by median rank (MedR) and recall (R@K). For example, using canonical ingredients with attention improved MedR on the 5K subset.
  • Enhanced Image Generation: CookGAN, particularly with the cycle-consistency constraint and canonical ingredients, generates more photo-realistic meal images. This is supported by quantitative metrics like Inception Score (IS) and Frechet Inception Distance (FID) on salad, cookie, and muffin subsets, where CookGAN generally outperforms StackGAN-v2.
  • Qualitative Improvements: Visual examples demonstrate that CookGAN produces images with clearer ingredient appearances and better overall realism compared to previous methods. The model can also effectively manipulate ingredient presence through linear interpolations in FoodSpace.

Limitations and Future Work

  • Difficulty of Meal Synthesis: The authors acknowledge that synthesizing photo-realistic meal images remains an extremely challenging task, and generated images still differ from real images in their retrieval ability.
  • Limited Context: The current model primarily relies on ingredient lists. Future work could incorporate additional information like recipe instructions and titles to provide richer context for meal preparation and appearance.
  • Ingredient Quantities: The model does not currently consider the quantities of ingredients, which could further enhance the realism and control over the synthesized images.
  • Focus on Specific Categories: For generation experiments, the authors focused on narrow meal categories (salad, cookie, muffin) where cooking methods are more consistent. Extending this to a wider range of complex dishes would be a valuable next step.

Summary updated

Abstract

In this work we propose a new computational framework, based on generative deep models, for synthesis of photo-realistic food meal images from textual list of its ingredients. Previous works on synthesis of images from text typically rely on pre-trained text models to extract text features, followed by generative neural networks (GAN) aimed to generate realistic images conditioned on the text features. These works mainly focus on generating spatially compact and well-defined categories of objects, such as birds or flowers, but meal images are significantly more complex, consisting of multiple ingredients whose appearance and spatial qualities are further modified by cooking methods. To generate real-like meal images from ingredients, we propose Cook Generative Adversarial Networks (CookGAN), CookGAN first builds an attention-based ingredients-image association model, which is then used to condition a generative neural network tasked with synthesizing meal images. Furthermore, a cycle-consistent constraint is added to further improve image quality and control appearance. Experiments show our model is able to generate meal images corresponding to the ingredients.
BibTeX
@inproceedings{han2020cookgan,
  author = {Fangda Han and Ricardo Guerrero and Vladimir Pavlovic},
  title = {{CookGAN: Meal Image Synthesis from Ingredients}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=ReGUxjQRP8W},
}
Figure