Abstract
This paper introduces CkNN (Cross-Modal k-Nearest-Neighbours), a simple yet powerful non-parametric baseline for matching food images to recipes. By decoupling the training of image and text encoders and using a lightweight alignment strategy, the authors not only expose the lack of strong baselines in the field but also set a new State-of-the-Art (SoTA) on the Recipe1M dataset by a significant margin.
Introduction
Cross-modal recipe retrieval—finding the exact recipe for a given food photo—is notoriously difficult. Food can look vastly different even when following the same recipe, and recipes often share similar ingredients while resulting in distinct dishes.
The authors identify two major "growing pains" in current research: a lack of strong baselines and "component confusion" caused by end-to-end training, which makes it difficult to isolate the contributions of image encoders, text encoders, and alignment logic. To address this, they propose a "Divide and Conquer" strategy that separates feature learning from modality alignment.
Methodology
Independent Encoders
Instead of training everything at once, image and text encoders are trained separately using a self-supervised classification task. "Noisy labels" (e.g., "chicken," "soup") are extracted from recipe titles and ingredients. The text model uses a simple Average Word Embedding (AWE) approach, while the image model utilizes standard ResNet-50 or ResNext-101 architectures.
CkNN: The Non-Parametric Baseline
To align the two modalities without complex optimization, the authors introduce CkNN. To find a recipe for a query image, the system finds the $k$ most similar images in the training set, retrieves their associated recipes, averages those embeddings to create a "proxy" representation in the text space, and finally searches the test set for the closest recipe.
Triplet Loss Alignment
To surpass the baseline, a lightweight alignment module is added: two simple feed-forward networks (one for each modality) trained with a standard triplet loss. This maps precomputed features into a joint 1024-dimensional space where matching pairs are pulled together and mismatches are pushed apart.
Experimental Results
The models were evaluated on the Recipe1M benchmark, specifically the challenging 10K test set where a query is ranked against 10,000 candidates.
As shown in the results table, the CkNN approach achieved a 19.1% Recall@1, outperforming the original Pic2Recipe (7.3%) and nearing the previous SoTA, ACME (22.9%), without any joint training. The full model (Triplet + AWE + ResNext) reached 30.0% Recall@1, a 30% relative improvement over previous best results. Furthermore, the alignment module is highly efficient, training in just 25 seconds per epoch on a single GPU.
Discussion and Insights
The results suggest that "simple is better." The CkNN baseline outperformed several sophisticated models using GANs and complex attention mechanisms. A key takeaway is that the quality of individual encoders (features) matters more than alignment complexity; swapping ResNet-50 for ResNext-101 led to massive performance gains. Decoupling training provides a more robust foundation for retrieval and serves as a "sanity check" for the field, highlighting that well-tuned simple methods can be more effective than over-complicated architectures.
Conclusion
While CkNN is heavily dependent on the diversity of training data and currently uses simple text encoding (AWE), it sets a high bar for future research. Future work could involve replacing AWE with modern architectures like Transformers or refining the joint embedding space with more sophisticated negative sampling strategies.
Summary updated