MenuAI: Restaurant Food Recommendation System via a Transformer-based Deep Learning Model

Xinwei Ju, Frank Po Wen Lo, Jianing Qiu, Peilun Shi, Jiachuan Peng, Benny LoarXiv 2022 · DOI · OpenReview

TL;DR

This paper introduces MenuAI, a novel restaurant food recommendation system that uses OCR and a Transformer-based deep learning model (Learning to Rank - LTR) to rank menu items based on specific nutritional needs. It addresses the challenge of dynamic user preferences and the need for context-aware recommendations, even for dishes not explicitly seen during training.

What Problem Does This Paper Address?

Making healthy dietary choices is crucial, especially for individuals managing chronic diseases. While eating at home allows for familiarity with ingredients, dining out presents a challenge due to the lack of in-depth understanding of menu items. Existing recommendation systems often struggle with the dynamic nature of food preferences and integrating nutritional knowledge. This paper tackles the problem of providing personalized food recommendations in restaurants by considering both user-specific nutritional needs and the context of the menu, even for dishes the system hasn't encountered before.

Methods

The MenuAI system comprises three main modules:

  1. Text Extraction Module: This module uses Optical Character Recognition (OCR) to digitize restaurant menus. It employs a Connectionist Text Proposal Network (CTPN) for text detection and a Convolutional Recurrent Neural Network (CRNN) for text recognition, converting menu images into digital text.
  2. Word Encoding Module: The extracted text is then encoded into feature vectors. Instead of traditional one-hot encoding, they use an index-based word encoding approach, creating "single word vectors" that are then grouped into "dish vectors" to represent each food item. Standardized dish names (max three words) and padding are used to ensure consistent input dimensions.
  3. Ranking Module (LTR Model): This core module utilizes a Transformer-based Learning to Rank (LTR) model. It takes the encoded dish vectors and a "search key" (e.g., calorie, protein level) as input. The model employs a self-attention mechanism, inspired by previous work, to facilitate context-aware learning by allowing each dish's representation to be influenced by all other dishes on the menu. This enables the system to learn a scoring function and infer the optimal ranking order for dishes based on the specified nutritional criteria. A key advantage is its ability to rank "unseen" food dishes.

Datasets

The authors introduce a self-constructed dataset called MenuRank. This dataset consists of 5625 menus, with each menu containing 7 to 15 food dishes. The ranking order of dishes within each menu was manually annotated based on a reputable nutritional dataset. For evaluation, the dataset was split into training and testing sets with an 80:20 ratio.

Key Insights and Ideas

  • Dynamic Preferences & Nutritional Needs: The paper highlights that food preferences are not static and that integrating nutritional knowledge is crucial for effective recommendations, especially for health-conscious individuals.
  • OCR for Menu Digitization: Leveraging OCR is a practical approach to bridge the gap between visual menus and digital recommendation systems, making the process more accessible.
  • Transformer for Contextual Ranking: The use of a Transformer-based LTR model with self-attention is a powerful way to capture inter-dish relationships on a menu, leading to more contextually relevant rankings.
  • Ranking Unseen Dishes: A significant contribution is the system's ability to rank food dishes that were not explicitly part of the training data, demonstrating a form of generalization.
  • Flexibility with Search Keys: The system's ability to rank dishes based on various nutritional criteria (calories, protein, etc.) via simple "search keys" makes it highly adaptable to different user needs.

Key Results

The MenuAI system demonstrated promising performance on the MenuRank dataset:

  • Single Search Key:
    • With 100% seen dishes, the system achieved an accuracy (ACC) of up to 99.5%.
  • Even with only 50% seen dishes, ACC was 86.5%, and with 10% seen dishes, it was 77.2%. These results validate the model's ability to handle unseen combinations and generalize.
  • Multiple Search Keys:
    • When trained with multiple search keys, the ACC dropped slightly but remained strong, reaching 96.1% for 100% seen dishes, 71.7% for 50% seen dishes, and 65.2% for 10% seen dishes. This indicates a trade-off but still shows good performance and the advantage of a single model for multiple criteria.

Limitations and Future Work

  • Dataset Size: The paper notes that only 38 different food servings were used in total for the MenuRank dataset, suggesting that scaling up the dataset could further improve performance.
  • BERT Integration: The authors mention that a BERT encoder will be applied in future work to facilitate text encoding, potentially enhancing the representation of food dish descriptions.
  • Performance with Multiple Keys: While promising, the performance with multiple search keys showed a slight drop, indicating room for improvement in training strategies for such scenarios.
  • Future Opportunities: The authors believe their research opens up new avenues for enhancing existing food recommendation systems by incorporating OCR and advanced deep learning techniques.

Summary updated

Abstract

Food recommendation system has proven as an effective technology to provide guidance on dietary choices, and this is especially important for patients suffering from chronic diseases. Unlike other multimedia recommendations, such as books and movies, food recommendation task is highly relied on the context at the moment, since users' food preference can be highly dynamic over time. For example, individuals tend to eat more calories earlier in the day and eat a little less at dinner. However, there are still limited research works trying to incorporate both current context and nutritional knowledge for food recommendation. Thus, a novel restaurant food recommendation system is proposed in this paper to recommend food dishes to users according to their special nutritional needs. Our proposed system utilises Optical Character Recognition (OCR) technology and a transformer-based deep learning model, Learning to Rank (LTR) model, to conduct food recommendation. Given a single RGB image of the menu, the system is then able to rank the food dishes in terms of the input search key (e.g., calorie, protein level). Due to the property of the transformer, our system can also rank unseen food dishes. Comprehensive experiments are conducted to validate our methods on a self-constructed menu dataset, known as MenuRank dataset. The promising results, with accuracy ranging from 77.2% to 99.5%, have demonstrated the great potential of LTR model in addressing food recommendation problems.
BibTeX
@misc{ju2022menuai,
  author = {Xinwei Ju and Frank Po Wen Lo and Jianing Qiu and Peilun Shi and Jiachuan Peng and Benny Lo},
  title = {{MenuAI: Restaurant Food Recommendation System via a Transformer-based Deep Learning Model}},
  year = {2022},
  eprint = {2210.08266},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.08266},
}
Figure