TL;DR
This paper introduces MenuAI, a novel restaurant food recommendation system that uses OCR and a Transformer-based deep learning model (Learning to Rank - LTR) to rank menu items based on specific nutritional needs. It addresses the challenge of dynamic user preferences and the need for context-aware recommendations, even for dishes not explicitly seen during training.
What Problem Does This Paper Address?
Making healthy dietary choices is crucial, especially for individuals managing chronic diseases. While eating at home allows for familiarity with ingredients, dining out presents a challenge due to the lack of in-depth understanding of menu items. Existing recommendation systems often struggle with the dynamic nature of food preferences and integrating nutritional knowledge. This paper tackles the problem of providing personalized food recommendations in restaurants by considering both user-specific nutritional needs and the context of the menu, even for dishes the system hasn't encountered before.
Methods
The MenuAI system comprises three main modules:
- Text Extraction Module: This module uses Optical Character Recognition (OCR) to digitize restaurant menus. It employs a Connectionist Text Proposal Network (CTPN) for text detection and a Convolutional Recurrent Neural Network (CRNN) for text recognition, converting menu images into digital text.
- Word Encoding Module: The extracted text is then encoded into feature vectors. Instead of traditional one-hot encoding, they use an index-based word encoding approach, creating "single word vectors" that are then grouped into "dish vectors" to represent each food item. Standardized dish names (max three words) and padding are used to ensure consistent input dimensions.
- Ranking Module (LTR Model): This core module utilizes a Transformer-based Learning to Rank (LTR) model. It takes the encoded dish vectors and a "search key" (e.g., calorie, protein level) as input. The model employs a self-attention mechanism, inspired by previous work, to facilitate context-aware learning by allowing each dish's representation to be influenced by all other dishes on the menu. This enables the system to learn a scoring function and infer the optimal ranking order for dishes based on the specified nutritional criteria. A key advantage is its ability to rank "unseen" food dishes.
Datasets
The authors introduce a self-constructed dataset called MenuRank. This dataset consists of 5625 menus, with each menu containing 7 to 15 food dishes. The ranking order of dishes within each menu was manually annotated based on a reputable nutritional dataset. For evaluation, the dataset was split into training and testing sets with an 80:20 ratio.
Key Insights and Ideas
- Dynamic Preferences & Nutritional Needs: The paper highlights that food preferences are not static and that integrating nutritional knowledge is crucial for effective recommendations, especially for health-conscious individuals.
- OCR for Menu Digitization: Leveraging OCR is a practical approach to bridge the gap between visual menus and digital recommendation systems, making the process more accessible.
- Transformer for Contextual Ranking: The use of a Transformer-based LTR model with self-attention is a powerful way to capture inter-dish relationships on a menu, leading to more contextually relevant rankings.
- Ranking Unseen Dishes: A significant contribution is the system's ability to rank food dishes that were not explicitly part of the training data, demonstrating a form of generalization.
- Flexibility with Search Keys: The system's ability to rank dishes based on various nutritional criteria (calories, protein, etc.) via simple "search keys" makes it highly adaptable to different user needs.
Key Results
The MenuAI system demonstrated promising performance on the MenuRank dataset:
- Single Search Key:
- With 100% seen dishes, the system achieved an accuracy (ACC) of up to 99.5%.
- Even with only 50% seen dishes, ACC was 86.5%, and with 10% seen dishes, it was 77.2%. These results validate the model's ability to handle unseen combinations and generalize.
- Multiple Search Keys:
- When trained with multiple search keys, the ACC dropped slightly but remained strong, reaching 96.1% for 100% seen dishes, 71.7% for 50% seen dishes, and 65.2% for 10% seen dishes. This indicates a trade-off but still shows good performance and the advantage of a single model for multiple criteria.
Limitations and Future Work
- Dataset Size: The paper notes that only 38 different food servings were used in total for the MenuRank dataset, suggesting that scaling up the dataset could further improve performance.
- BERT Integration: The authors mention that a BERT encoder will be applied in future work to facilitate text encoding, potentially enhancing the representation of food dish descriptions.
- Performance with Multiple Keys: While promising, the performance with multiple search keys showed a slight drop, indicating room for improvement in training strategies for such scenarios.
- Future Opportunities: The authors believe their research opens up new avenues for enhancing existing food recommendation systems by incorporating OCR and advanced deep learning techniques.
Summary updated