CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval

Hai Xuan Pham, Ricardo Guerrero, Vladimir Pavlovic, Jiatong Li

Abstract

Despite the abundance of multi-modal data, such as image-text pairs, there has been little effort in understanding the individual entities and their different roles in the construction of these data instances. In this work, we endeavour to discover the entities and their corresponding importance in cooking recipes automatically as a visual-linguistic association problem. More specifically, we introduce a novel cross-modal learning framework to jointly model the latent representations of images and text in the food image-recipe association and retrieval tasks. This model allows one to discover complex functional and hierarchical relationships between images and text, and among textual parts of a recipe including title, ingredients and cooking instructions. Our experiments show that by making use of efficient tree-structured Long Short-Term Memory as the text encoder in our computational cross-modal retrieval framework, we are not only able to identify the main ingredients and cooking actions in the recipe descriptions without explicit supervision, but we can also learn more meaningful feature representations of food recipes, appropriate for challenging cross-modal retrieval and recipe adaption tasks.

Introduction

Cross-modal retrieval—the task of finding a recipe from a photograph or vice versa—presents unique challenges in the food domain. Unlike general object recognition, a cooked dish is a complex transformation of various ingredients. Traditional models often treat recipes as simple linear sequences of words, failing to capture the inherent hierarchy where a title summarizes a dish, ingredients are combined in specific proportions, and instructions follow a logical flow. The CHEF (Cross-modal Hierarchical Embeddings for Food) framework addresses this by modeling the functional relationships between recipe components, distinguishing between "main" ingredients and "minor" ones, and understanding how cooking actions (e.g., grilling vs. boiling) alter visual representations.

Methodology

The core of the CHEF framework is a novel cross-modal learning architecture that aligns textual recipes with visual data in a shared latent space.

The CHEF framework architecture, showing the joint embedding of recipe components (title, ingredients, instructions) and visual features.

Hierarchical Text Encoding

Instead of standard linear LSTMs, CHEF utilizes Tree-structured LSTMs for text encoding. The model employs a Gumbel-softmax Tree-LSTM to learn the most useful hierarchical structure directly from the data without explicit supervision. This allows the model to process recipes at multiple levels: word-to-sentence and sentence-to-instruction, separately encoding the title, ingredient list, and cooking steps.

Joint Latent Space and Training

Textual features from the Tree-LSTM and visual features from a ResNet50-based CNN are projected into a shared 1024-dimensional space. The system is trained using a triplet loss function, which minimizes the distance between matching image-recipe pairs while maximizing the distance for non-matching pairs.

Dataset and Preprocessing

The model was trained and evaluated on the Recipe1M (R1M) dataset, containing approximately 1 million recipes and 0.5 million images. A "Canonical Ingredient" list was created to standardize ~16k unique ingredient strings into ~1.4k terms using stemming and word2vec clustering.

Hierarchical Importance and Discovery

A key innovation of CHEF is its ability to automatically identify the importance of different entities within a recipe.

An example of a learned hierarchical tree for ingredients, where proximity to the root indicates higher importance.

In the learned ingredient trees, ingredients positioned closer to the "root" tend to be the most visually and functionally significant. For instance, the model naturally learns that "chicken" is more central to a dish's identity than "salt." Experiments showed that removing "least important" ingredients (those furthest from the root) initially improved retrieval performance, suggesting the model effectively filters out visual "noise." Furthermore, the model identifies key cooking verbs (e.g., "bake," "stir") as influential nodes, with over 76% of the most important leaf nodes in instruction trees being verbs.

Results and Discussion

CHEF achieves state-of-the-art performance in cross-modal retrieval tasks. On a 10k test set, CHEF achieved a median Rank (medR) of 7.3, significantly outperforming previous models like ACME (medR 9.4).

The model also demonstrated strong capabilities in:

  • Main Ingredient Detection: Correctly identifying the "true" main ingredient in 47% of cases, far exceeding random chance (~11%).
  • Recipe Surgery: Successfully performing ingredient substitutions (e.g., swapping "chicken" for "beef") and retrieving the corresponding modified dish images.

Limitations

While effective, the hierarchical approach faced challenges with full instruction sets. The authors found that a linear sequence was often better for entire instruction blocks because cooking steps are strictly chronological. Future work aims to refine the relationship between specific visual regions and textual entities to further improve recipe adaptation and dietary modification tasks.

Summary updated

Abstract

Despite the abundance of multi-modal data, such as image-text pairs, there has been little effort in understanding the individual entities and their different roles in the construction of these data instances. In this work, we endeavour to discover the entities and their corresponding importance in cooking recipes automatically as a visual-linguistic association problem. More specifically, we introduce a novel cross-modal learning framework to jointly model the latent representations of images and text in the food image-recipe association and retrieval tasks. This model allows one to discover complex functional and hierarchical relationships between images and text, and among textual parts of a recipe including title, ingredients and cooking instructions. Our experiments show that by making use of efficient tree-structured Long Short-Term Memory as the text encoder in our computational cross-modal retrieval framework, we are not only able to identify the main ingredients and cooking actions in the recipe descriptions without explicit supervision, but we can also learn more meaningful feature representations of food recipes, appropriate for challenging cross-modal retrieval and recipe adaption tasks.
BibTeX
@inproceedings{pham2021chef,
  author = {Hai Xuan Pham and Ricardo Guerrero and Vladimir Pavlovic and Jiatong Li},
  title = {{CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=_huAj1H_idBO},
}
Figure