Cross-domain Food Image-to-Recipe Retrieval by Weighted Adversarial Learning

Bin Zhu, Chong-Wah Ngo, Jingjing Chen, Wing-Kwong ChanarXiv 2023 · DOI · arXiv

Introduction

Food image-to-recipe retrieval is a complex task that typically requires large-scale paired datasets of images and their corresponding recipes. However, most existing systems are "monocultural," performing well only when the training and query data originate from the same cuisine. Collecting paired data for every regional cuisine is often impractical due to data scarcity or privacy constraints. This paper addresses the challenge of unsupervised domain adaptation, where a model trained on a data-rich "source" cuisine (e.g., Sichuan) must retrieve recipes for a "target" cuisine (e.g., Japanese) that lacks paired image-recipe training data.

Methodology: Bridging the Cuisine Gap

The authors propose a novel framework centered on two key mechanisms: the Source Batch Selector (SBS) and Weighted Cross-modal Adversarial Learning.

Source Batch Selector (SBS)

Rather than utilizing all available source data, which can introduce noise if the dishes are too distinct from the target domain, the SBS "cherry-picks" samples. It uses a pre-trained recipe encoder to identify source recipes most similar to the target domain's recipes. This filtering process ensures the model focuses on transferable culinary features and avoids "disturbances" from source-specific dishes that would otherwise degrade performance.

Weighted Adversarial Learning

To further refine the adaptation, the model assigns a similarity weight ($W_2$) to each source sample. This weight influences two primary loss functions:

  1. Weighted Triplet Loss: Prioritizes image-recipe pairs that closely resemble the target cuisine in the embedding space.
  2. Weighted Adversarial Loss: Trains a recipe encoder to "fool" a domain discriminator. By weighting this loss, the model prioritizes the alignment of relevant samples over a forced, global distribution overlap.
The weighting mechanism assigns lower weights to source samples that are less relevant to the target domain, reducing their influence on the adaptation process.

Experimental Evaluation

The framework was tested across three major cuisines: Sichuan (Chuan), Cantonese (Yue), and Japanese (Washoku). The dataset included over 79,000 recipes and 320,000 images in total. To facilitate mapping, the authors utilized a shared vocabulary of 1,635 common ingredients across these cuisines. The architecture utilized a ResNet-50 backbone for images and a hierarchical LSTM for recipes, processing titles, ingredients, and instructions as distinct semantic components.

Key Results

Performance was evaluated using Median Rank (MedR), where a lower score indicates better retrieval accuracy.

  • Superior Performance: The proposed method achieved an average MedR of 14.5, significantly outperforming standard adversarial networks like IWAN (MedR 23.6) and the previous state-of-the-art CCFT (MedR 15.9).
  • Cuisine-Specific Gains: In the transfer from Cantonese to Japanese, the model improved the MedR from 15.7 to 13.1.
  • Effectiveness of SBS: Ablation studies confirmed that removing the Source Batch Selector led to a significant drop in performance, highlighting the importance of intelligent data selection.
  • The Oracle Gap: While the cross-domain approach is state-of-the-art, a significant performance gap remains compared to an "Oracle" model trained on fully paired target data (MedR 5.5), indicating potential for future refinement.

Conclusion and Future Work

This research demonstrates that recipes serve as an effective "anchor" for domain transfer due to their stable semantic content (ingredients and instructions) compared to visual pixels. By intelligently filtering and weighting source data, the model successfully bridges the gap between disparate culinary traditions. Future work aims to address fine-grained distinctions between similar recipes and explore cross-lingual retrieval to further expand the system's global applicability.

Summary updated

BibTeX
@misc{zhu2023cross,
  author = {Bin Zhu and Chong-Wah Ngo and Jingjing Chen and Wing-Kwong Chan},
  title = {{Cross-domain Food Image-to-Recipe Retrieval by Weighted Adversarial Learning}},
  year = {2023},
  eprint = {2304.07387},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.07387},
}
Figure