Abstract
Despite the abundance of multi-modal data, such as image-text pairs, there has been little effort in understanding the individual entities and their different roles in the construction of these data instances. In this work, we endeavour to discover the entities and their corresponding importance in cooking recipes automatically as a visual-linguistic association problem. More specifically, we introduce a novel cross-modal learning framework to jointly model the latent representations of images and text in the food image-recipe association and retrieval tasks. This model allows one to discover complex functional and hierarchical relationships between images and text, and among textual parts of a recipe including title, ingredients and cooking instructions. Our experiments show that by making use of efficient tree-structured Long Short-Term Memory as the text encoder in our computational cross-modal retrieval framework, we are not only able to identify the main ingredients and cooking actions in the recipe descriptions without explicit supervision, but we can also learn more meaningful feature representations of food recipes, appropriate for challenging cross-modal retrieval and recipe adaption tasks.
Introduction
Cross-modal retrieval—the task of finding a recipe from a photograph or vice versa—presents unique challenges in the food domain. Unlike general object recognition, a cooked dish is a complex transformation of various ingredients. Traditional models often treat recipes as simple linear sequences of words, failing to capture the inherent hierarchy where a title summarizes a dish, ingredients are combined in specific proportions, and instructions follow a logical flow. The CHEF (Cross-modal Hierarchical Embeddings for Food) framework addresses this by modeling the functional relationships between recipe components, distinguishing between "main" ingredients and "minor" ones, and understanding how cooking actions (e.g., grilling vs. boiling) alter visual representations.
Methodology
The core of the CHEF framework is a novel cross-modal learning architecture that aligns textual recipes with visual data in a shared latent space.
Hierarchical Text Encoding
Instead of standard linear LSTMs, CHEF utilizes Tree-structured LSTMs for text encoding. The model employs a Gumbel-softmax Tree-LSTM to learn the most useful hierarchical structure directly from the data without explicit supervision. This allows the model to process recipes at multiple levels: word-to-sentence and sentence-to-instruction, separately encoding the title, ingredient list, and cooking steps.
Joint Latent Space and Training
Textual features from the Tree-LSTM and visual features from a ResNet50-based CNN are projected into a shared 1024-dimensional space. The system is trained using a triplet loss function, which minimizes the distance between matching image-recipe pairs while maximizing the distance for non-matching pairs.
Dataset and Preprocessing
The model was trained and evaluated on the Recipe1M (R1M) dataset, containing approximately 1 million recipes and 0.5 million images. A "Canonical Ingredient" list was created to standardize ~16k unique ingredient strings into ~1.4k terms using stemming and word2vec clustering.
Hierarchical Importance and Discovery
A key innovation of CHEF is its ability to automatically identify the importance of different entities within a recipe.
In the learned ingredient trees, ingredients positioned closer to the "root" tend to be the most visually and functionally significant. For instance, the model naturally learns that "chicken" is more central to a dish's identity than "salt." Experiments showed that removing "least important" ingredients (those furthest from the root) initially improved retrieval performance, suggesting the model effectively filters out visual "noise." Furthermore, the model identifies key cooking verbs (e.g., "bake," "stir") as influential nodes, with over 76% of the most important leaf nodes in instruction trees being verbs.
Results and Discussion
CHEF achieves state-of-the-art performance in cross-modal retrieval tasks. On a 10k test set, CHEF achieved a median Rank (medR) of 7.3, significantly outperforming previous models like ACME (medR 9.4).
The model also demonstrated strong capabilities in:
- Main Ingredient Detection: Correctly identifying the "true" main ingredient in 47% of cases, far exceeding random chance (~11%).
- Recipe Surgery: Successfully performing ingredient substitutions (e.g., swapping "chicken" for "beef") and retrieving the corresponding modified dish images.
Limitations
While effective, the hierarchical approach faced challenges with full instruction sets. The authors found that a linear sequence was often better for entire instruction blocks because cooking steps are strictly chronological. Future work aims to refine the relationship between specific visual regions and textual entities to further improve recipe adaptation and dietary modification tasks.
Summary updated