Leveraging Context to Support Automated Food Recognition in Restaurants

Vinay Bettadapura, Edison Thomaz, Aman Parnami, Gregory D. Abowd, Irfan A. Essa

TL;DR

This paper presents an automated food recognition system that drastically simplifies the computer vision challenge of identifying meals "in the wild" by leveraging smartphone location data. By using GPS to identify the specific restaurant a user is in, the system automatically pulls the online menu, scrapes Google Images for training data, and builds a highly constrained, on-the-fly classifier to identify the user's meal.

What Problem Does This Paper Address?

Dietary self-monitoring (food journaling) is a proven method for weight loss and meeting nutritional goals, but manual logging is tedious, time-consuming, and prone to under-reporting. While people are increasingly taking photos of their food, automatically recognizing what is in those photos is a massive computer vision challenge. Food has extreme intra-class variability (a burger at McDonald's looks different than a burger at a pub) and there are thousands of potential food categories.

Previous automated approaches have relied on clunky wearable sensors (like acoustic neckbands to detect swallowing), expensive and slow crowdsourcing (like Amazon Mechanical Turk), or highly constrained lab-environment image recognition. This paper tackles the gap by proposing a fully automated, scalable computer vision pipeline that uses contextual metadata (location) to turn an impossible "recognize any food" problem into a highly solvable "recognize which of these 15 menu items this is" problem.

Methods

The authors propose a clever pipeline that combines contextual sensor data with weakly supervised machine learning:

  1. Contextual Localization: The system extracts the geo-tag (GPS coordinates) from a smartphone or wearable camera image and queries the Yelp or Google Places API to identify the specific restaurant.
  2. Weakly Supervised Training: Once the restaurant is identified, the system scrapes its online menu. For each menu item, it automatically downloads the top 50 images from Google Image Search to serve as weakly-labeled training data.
  3. Image Segmentation: Test images are processed using hierarchical segmentation to isolate the actual food from the background (plates, tables).
  4. Feature Extraction: Because food varies wildly in both shape and lighting, the authors extract a robust mix of six feature descriptors: two color-based (Color Moment Invariants, Hue Histograms) and four SIFT-based (C-SIFT, OpponentSIFT, RGB-SIFT, standard SIFT). These are mapped into a Bag-of-Words (BoW) representation.
  5. Classification: They use a Sequential Minimal Optimization Multiple Kernel Learning (SMO-MKL) Support Vector Machine. Essentially, MKL allows the model to optimally combine the different color and texture features to make a final prediction.

Datasets

  • PFID (Pittsburgh Fast-Food Image Dataset): An existing benchmark dataset of 61 fast-food categories captured in controlled lab conditions. Used to compare the proposed feature extraction/MKL approach against existing state-of-the-art models.
  • "In-the-Wild" Restaurant Dataset (Newly Introduced): 600 images collected by three individuals using smartphones and Google Glass across 10 real-world restaurants spanning 5 cuisines (American, Indian, Italian, Mexican, Thai). The training set consisted of 3,750 weakly-labeled images scraped from Google Image Search (15 dishes per cuisine x 50 images).

Key Insights and Ideas

  • Context is a massive force multiplier for CV: The "aha moment" of the paper is quantifying exactly how much location data helps. By restricting the classifier's search space to just the items on a specific restaurant's menu, the classification accuracy jumped from a dismal 15.67% to 63.33%.
  • Food requires both color and texture: Standard object recognition often relies heavily on shape/texture (like SIFT). However, food recognition requires a blend. The MKL approach proved that dynamically combining color descriptors (to handle lighting changes) and SIFT variants yields significantly better results than any single descriptor.
  • The web as an automated annotator: By using Google Image Search based on menu text, the system completely bypasses the need for manually labeled training data, making the system infinitely scalable to new restaurants.

Key Results

  • Shattering the PFID Baseline: On the standard PFID benchmark, the authors' SMO-MKL approach achieved 48.5% accuracy, absolutely crushing the previous state-of-the-art of 28.9% (a >20% absolute improvement).
  • Real-World Performance: On the custom "in-the-wild" dataset, the system achieved an overall average accuracy of 63.33%.
  • Cuisine Breakdown: Accuracy varied significantly by cuisine. Indian food was recognized with the highest accuracy (80.8%), while Mexican (43.3%) and Thai (50.8%) proved the most difficult.

Limitations and Future Work

  • Intra-Cuisine Similarity: The system struggled with Mexican and Thai foods because dishes within those cuisines often look visually identical (e.g., a burrito vs. an enchilada, or Basil Fried Rice vs. Mandarin Fried Rice).
  • Location Ambiguity: The pipeline assumes GPS can perfectly identify the restaurant. In dense areas like shopping malls or food courts, GPS might pull the wrong menu entirely.
  • Generic Training Data: Training on generic Google Images of "Pad Thai" isn't perfectly representative of what a specific restaurant's Pad Thai looks like.
  • Future Work: The authors suggest a brilliant next step: instead of scraping generic Google Images, the system could scrape crowdsourced photos from Yelp for that exact restaurant, training a highly specialized classifier on the actual dishes prepared by that specific kitchen.

Summary updated

Abstract

The pervasiveness of mobile cameras has resulted in a dramatic increase in food photos, which are pictures reflecting what people eat. In this paper, we study how taking pictures of what we eat in restaurants can be used for the purpose of automating food journaling. We propose to leverage the context of where the picture was taken, with additional information about the restaurant, available online, coupled with state-of-the-art computer vision techniques to recognize the food being consumed. To this end, we demonstrate image-based recognition of foods eaten in restaurants by training a classifier with images from restaurant's online menu databases. We evaluate the performance of our system in unconstrained, real-world settings with food images taken in 10 restaurants across 5 different types of food (American, Indian, Italian, Mexican and Thai).
BibTeX
@inproceedings{bettadapura2015leveraging,
  author = {Vinay Bettadapura and Edison Thomaz and Aman Parnami and Gregory D. Abowd and Irfan A. Essa},
  title = {{Leveraging Context to Support Automated Food Recognition in Restaurants}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=AsCZg2uVR0},
}

Loading related content...

Figure