Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA

Mikhail Fain, Andrey Ponikar, Ryan Fox, Danushka BollegalaarXiv 2019 · arXiv

Abstract

This paper introduces CkNN (Cross-Modal k-Nearest-Neighbours), a simple yet powerful non-parametric baseline for matching food images to recipes. By decoupling the training of image and text encoders and using a lightweight alignment strategy, the authors not only expose the lack of strong baselines in the field but also set a new State-of-the-Art (SoTA) on the Recipe1M dataset by a significant margin.

Introduction

Cross-modal recipe retrieval—finding the exact recipe for a given food photo—is notoriously difficult. Food can look vastly different even when following the same recipe, and recipes often share similar ingredients while resulting in distinct dishes.

The authors identify two major "growing pains" in current research: a lack of strong baselines and "component confusion" caused by end-to-end training, which makes it difficult to isolate the contributions of image encoders, text encoders, and alignment logic. To address this, they propose a "Divide and Conquer" strategy that separates feature learning from modality alignment.

Methodology

Independent Encoders

Instead of training everything at once, image and text encoders are trained separately using a self-supervised classification task. "Noisy labels" (e.g., "chicken," "soup") are extracted from recipe titles and ingredients. The text model uses a simple Average Word Embedding (AWE) approach, while the image model utilizes standard ResNet-50 or ResNext-101 architectures.

CkNN: The Non-Parametric Baseline

To align the two modalities without complex optimization, the authors introduce CkNN. To find a recipe for a query image, the system finds the $k$ most similar images in the training set, retrieves their associated recipes, averages those embeddings to create a "proxy" representation in the text space, and finally searches the test set for the closest recipe.

The CkNN process for mapping a textual document into the image embedding space.

Triplet Loss Alignment

To surpass the baseline, a lightweight alignment module is added: two simple feed-forward networks (one for each modality) trained with a standard triplet loss. This maps precomputed features into a joint 1024-dimensional space where matching pairs are pulled together and mismatches are pushed apart.

Experimental Results

The models were evaluated on the Recipe1M benchmark, specifically the challenging 10K test set where a query is ranked against 10,000 candidates.

Retrieval performance comparison on Recipe1M 1K and 10K test sets.

As shown in the results table, the CkNN approach achieved a 19.1% Recall@1, outperforming the original Pic2Recipe (7.3%) and nearing the previous SoTA, ACME (22.9%), without any joint training. The full model (Triplet + AWE + ResNext) reached 30.0% Recall@1, a 30% relative improvement over previous best results. Furthermore, the alignment module is highly efficient, training in just 25 seconds per epoch on a single GPU.

Discussion and Insights

The results suggest that "simple is better." The CkNN baseline outperformed several sophisticated models using GANs and complex attention mechanisms. A key takeaway is that the quality of individual encoders (features) matters more than alignment complexity; swapping ResNet-50 for ResNext-101 led to massive performance gains. Decoupling training provides a more robust foundation for retrieval and serves as a "sanity check" for the field, highlighting that well-tuned simple methods can be more effective than over-complicated architectures.

Conclusion

While CkNN is heavily dependent on the diversity of training data and currently uses simple text encoding (AWE), it sets a high bar for future research. Future work could involve replacing AWE with modern architectures like Transformers or refining the joint embedding space with more sophisticated negative sampling strategies.

Summary updated

BibTeX
@misc{fain2019dividing,
  author = {Mikhail Fain and Andrey Ponikar and Ryan Fox and Danushka Bollegala},
  title = {{Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA}},
  year = {2019},
  eprint = {1911.12763},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.12763},
}
Figure