Abstract
The pervasiveness of mobile cameras has resulted in a dramatic increase in food photos, which are pictures reflecting what people eat. In this paper, we study how taking pictures of what we eat in restaurants can be used for the purpose of automating food journaling. We propose to leverage the context of where the picture was taken, with additional information about the restaurant, available online, coupled with state-of-the-art computer vision techniques to recognize the food being consumed. To this end, we demonstrate image-based recognition of foods eaten in restaurants by training a classifier with images from restaurant's online menu databases. We evaluate the performance of our system in unconstrained, real-world settings with food images taken in 10 restaurants across 5 different types of food (American, Indian, Italian, Mexican and Thai).
Introduction
Dietary self-monitoring is a cornerstone of weight management and nutritional health, yet manual logging remains a significant barrier for most users due to its tedious and time-consuming nature. While the rise of smartphone photography offers a potential solution, automated food recognition "in the wild" remains an immense computer vision challenge. Food items exhibit extreme intra-class variability and often lack the rigid structures found in other object categories, making generic classification difficult.
This research addresses these challenges by leveraging contextual metadata—specifically GPS location—to constrain the problem space. By identifying the specific restaurant where a photo is taken, the system transforms an open-ended recognition task into a highly solvable closed-set classification problem based on that restaurant's specific menu.
The Automated Recognition Pipeline
The proposed system utilizes a multi-stage pipeline that integrates sensor data with weakly supervised learning to identify meals without manual user intervention.
- Contextual Localization: The system extracts GPS coordinates from the image geo-tags and queries APIs (such as Yelp or Google Places) to identify the specific restaurant.
- Automated Training Set Generation: Once the restaurant is identified, the system scrapes its online menu. For each menu item, it automatically downloads approximately 50 images from Google Image Search to serve as weakly-labeled training data.
- Image Segmentation and Feature Extraction: Test images undergo hierarchical segmentation to isolate the food from background elements like plates and tables. The system then extracts a robust set of features, including color descriptors (Color Moment Invariants, Hue Histograms) and multiple SIFT variants (C-SIFT, OpponentSIFT, RGB-SIFT).
- Classification: A Sequential Minimal Optimization Multiple Kernel Learning (SMO-MKL) Support Vector Machine (SVM) is employed to optimally combine these diverse features for the final prediction.
Experimental Evaluation
The system was evaluated using two primary datasets:
- PFID (Pittsburgh Fast-Food Image Dataset): A benchmark of 61 fast-food categories captured in controlled conditions, used to validate the feature extraction and MKL approach against existing state-of-the-art models.
- "In-the-Wild" Restaurant Dataset: A new collection of 600 images gathered by individuals using smartphones and Google Glass across 10 real-world restaurants spanning five cuisines (American, Indian, Italian, Mexican, and Thai).
Results and Discussion
The inclusion of location context proved to be a massive force multiplier for recognition accuracy. By restricting the classifier's search space to a specific restaurant's menu, classification accuracy on the real-world dataset jumped from a baseline of 15.67% to 63.33%.
On the standard PFID benchmark, the SMO-MKL approach achieved 48.5% accuracy, representing a significant absolute improvement over the previous state-of-the-art of 28.9%. In real-world testing, performance varied by cuisine; Indian food achieved the highest recognition rate (80.8%), while Mexican (43.3%) and Thai (50.8%) proved more difficult due to high visual similarity between different dishes within those cuisines (e.g., burritos vs. enchiladas).
Conclusion and Future Work
By combining contextual sensor data with automated web-based training, this system provides a scalable path toward fully automated food journaling. While the current system uses generic web images for training, future iterations could leverage crowdsourced, restaurant-specific photos from platforms like Yelp to train even more specialized classifiers that account for the unique presentation styles of individual kitchens.
Summary updated