Deep-based Ingredient Recognition for Cooking Recipe Retrieval

Jingjing Chen, Chong-Wah NgoACM Multimedia 2016 · DOI

TL;DR

This paper tackles the challenge of retrieving cooking recipes from dish images, especially for dishes with unknown categories. They propose a multi-task deep learning approach that simultaneously learns to recognize food categories and their constituent ingredients. This ingredient-focused approach enables zero-shot retrieval, allowing recipe lookup even when the dish's category is unfamiliar.

What Problem Does This Paper Address?

Finding the right cooking recipe from a picture of a dish is a common need, particularly for applications like dietary monitoring. Existing methods often focus on recognizing the global appearance of a dish to identify its category. However, this approach fails when encountering dishes from categories not seen during training (zero-shot retrieval). Furthermore, relying solely on food categories is problematic for cuisines like Chinese food, where dishes within the same category can have vastly different ingredients and presentations. The paper argues that recognizing individual ingredients is a more scalable and robust approach for recipe retrieval, especially for zero-shot scenarios, as the number of ingredients is far smaller than the number of food categories. However, ingredient recognition itself is a harder task than food categorization due to visual variations.

Methods

The core of the proposed solution is a multi-task deep learning framework that jointly learns food categorization and ingredient recognition. This is achieved by modifying a VGG-16 network. The authors explore four different deep architectures (Arch-A to Arch-D) that vary in how the two tasks share information and layers. Arch-D, which shares some convolutional layers and has private layers for each task, emerges as the best performing architecture.

For ingredient recognition, which is a multi-label problem, a cross-entropy loss function is used. For food categorization, a multinomial logistic loss function is employed. The overall loss is a weighted sum of these two.

To enhance ingredient recognition and enable zero-shot retrieval, a Conditional Random Field (CRF) is introduced. This CRF models the contextual relationships (co-occurrences) between ingredients, learned from a large corpus of recipes. This helps refine the predicted ingredient probabilities, reducing errors, especially for unseen food categories.

Finally, for recipe retrieval, a query image is represented by its predicted ingredient probabilities (refined by CRF). This representation is then matched against ingredient lists from a recipe corpus using a similarity score.

Datasets

The primary dataset used is VIREO Food-172, a newly collected dataset specifically for Chinese food. It contains 172 food categories and 353 ingredient labels, with 110,241 images. This dataset is notable for including ingredient labels, which are often missing in existing food datasets like Food-101, UEC Food-100, and PFID. The authors also conducted experiments on the UEC Food-100 dataset for Japanese dishes to validate their approach on a different cuisine. A large recipe corpus of over 60,000 recipes was also collected for training the CRF and for recipe retrieval.

Key Insights and Ideas

  • Ingredient-centric retrieval is key for zero-shot: The paper strongly advocates for ingredient recognition as a more scalable and effective approach for recipe retrieval, especially when dealing with unknown food categories, compared to relying solely on food category recognition.
  • Multi-task learning boosts performance: Jointly learning food categorization and ingredient recognition leads to better performance for both tasks, particularly for ingredient recognition, compared to single-task learning. This suggests a beneficial mutual influence between the tasks.
  • Contextual ingredient information matters: Using a CRF to model ingredient co-occurrences from a large recipe corpus significantly refines ingredient predictions, improving accuracy and enabling more robust zero-shot retrieval.
  • Deep features are powerful for retrieval: Features extracted from the specialized layers of the multi-task network (Arch-D) show superior performance in zero-shot recipe retrieval compared to traditional hand-crafted features and even features from a single-task VGG network.

Key Results

  • The proposed multi-task deep architecture (Arch-D) achieved the best performance in both food categorization (Top-1 accuracy of 82.06%) and ingredient recognition (Micro-F1 of 67.17%) on the VIREO Food-172 dataset, outperforming single-task VGG and other multi-task variants.
  • The CRF-based ingredient refinement improved the Micro-F1 score for ingredient recognition from 63.94% to 66.23% for single-task learning and from 68.84% to 71.25% for multi-task learning.
  • In zero-shot recipe retrieval experiments, the fused approach combining image retrieval and ingredient matching using Arch-D achieved the highest top-10 hit rate of 0.570, significantly outperforming individual components and baselines.
  • The ingredient recognition performance was statistically significantly improved by Arch-D compared to VGG, with p-values close to 0.

Limitations and Future Work

  • Observable Ingredients: The current approach primarily relies on visible ingredients. Ingredients that are not directly observable (e.g., honey, soybean oil) or are part of a sauce are not explicitly handled.
  • Unseen Ingredients: While the zero-shot retrieval handles unseen food categories, the problem of dealing with unseen ingredients remains a challenge.
  • Cooking Methods and Ingredient Preparation: The model does not explicitly consider cooking methods (e.g., frying, steaming) or how ingredients are prepared (e.g., sliced, minced), which can lead to significant visual differences in dishes with the same ingredients.
  • Future Work: The authors suggest exploring hierarchical ingredient recognition and developing methods to handle unseen ingredients. They also propose learning an embedded space that captures the "translation" between dish images and textual recipes to better deal with unknown food and ingredient labels.

Summary updated

BibTeX
@inproceedings{chen2016deep,
  author = {Jingjing Chen and Chong-Wah Ngo},
  title = {{Deep-based Ingredient Recognition for Cooking Recipe Retrieval}},
  booktitle = {ACM Multimedia},
  year = {2016},
  doi = {10.1145/2964284.2964315},
}
Figure