I Ate This: A Photo-based Food Journaling System with Expert Feedback.

Shubham Goyal, Qi Liu, Khairina Tajul-Arifin, Waqas Awan, Bimlesh Wadhwa, Zhenguang LiuarXiv 2017 · OpenReview

TL;DR

This paper introduces "I Ate This," a novel photo-based food journaling system that bridges the gap between user-friendly food logging and expert nutritional guidance. By combining user-submitted food photos with dietitian evaluations, the system aims to provide personalized feedback, promote healthier eating habits, and ultimately drive long-term behavior change.

What Problem Does This Paper Address?

Accurate and consistent food journaling is a cornerstone of many health and wellness goals, from weight management to diabetes control. However, traditional methods of manual logging are often burdensome, time-consuming, and prone to inaccuracies due to difficulties in identifying food items, estimating portions, and recalling details. Existing smartphone apps, while improving accessibility, still struggle with user engagement and data reliability, leading to low adherence rates. This paper addresses the challenge of making food journaling both convenient and effective by leveraging visual information and expert human feedback.

Methods

The core of the "I Ate This" system is inspired by the Remote Food Photography Method (RFPM). The system comprises two main components:

  • Smartphone App (User-facing): Users capture a photo of their meal and provide a basic text description, optionally adding details like portion size, time, and self-rated healthiness. This lightweight capture process aims to minimize user effort.
  • Back Office System (Dietitian-facing): This system allows qualified dietitians to review submitted photos. The evaluation process involves three key steps:
  • Match: Dietitians match the photo to entries in a food database and estimate portion sizes.
  • Tag: Photos are tagged with nutritional information (e.g., Good Carb/Poor Carb, High Sugar/Low Sugar) and estimated fruit/vegetable servings.
  • Score: A final score from "Very Unhealthy" to "Very Healthy" is assigned.

The system also supports two levels of feedback: quantitative (algorithmic alerts based on aggregated data) and qualitative (personalized messages and encouragement from dietitians). Furthermore, the system is designed with APIs for integration into other health and web applications, and it can be integrated with Continuous Glucose Monitoring (CGM) devices for automated feedback tailored to individuals with diabetes.

Datasets

Not explicitly discussed in the paper. The system relies on a proprietary Holmusk nutrition database and image database.

Key Insights and Ideas

  • Bridging the Human-AI Gap in Food Journaling: While automated food recognition is improving, it's not yet accurate enough for reliable real-world use. "I Ate This" ingeniously combines the ease of photo capture with the nuanced understanding of human experts (dietitians) to overcome this limitation.
  • The Power of Lightweight, Visual Logging: The success of RFPM and its adoption in "I Ate This" highlights that simplifying the input process (photo + minimal text) significantly lowers the barrier to entry for food journaling, potentially increasing user adherence.
  • Personalized Feedback Drives Behavior Change: The system's emphasis on qualitative, personalized feedback from dietitians, in addition to quantitative data, is crucial for nudging users towards better choices and fostering long-term habit modification.
  • Flexibility for Diverse Health Needs: The system's architecture, including the potential for integration with CGM devices and restaurant databases, demonstrates its adaptability for various user groups and health goals, particularly for managing conditions like diabetes.

Key Results

The paper focuses on the system's design and proposed functionality rather than presenting quantitative evaluation results. The key "results" are the system's capabilities:

  • Convenient Food Logging: Users can log meals with a photo and basic description.
  • Expert Evaluation: Dietitians can provide detailed nutritional tagging and scoring.
  • Dual Feedback Mechanism: Offers both automated alerts and personalized dietitian messages.
  • Potential for Integration: Designed with APIs for broader application and CGM integration for diabetes management.

Limitations and Future Work

  • Scalability of Expert Feedback: While the system aims to be cheaper than face-to-face consultations, relying on human dietitians for evaluation might present scalability challenges as the user base grows.
  • Accuracy of Automated Components: The paper acknowledges that automated food recognition is still in its infancy and not sufficiently accurate for real-world use, necessitating human intervention.
  • Future Work: The authors plan to validate the "I Ate This" system for diabetes patients, integrate with CGM devices to color-code food photos based on their impact on blood glucose, and build a database of restaurant meals for easier logging. They also aim to make the system's APIs widely available for integration into other apps.

Summary updated

Abstract

What we eat is one of the most frequent and important health decisions we make in daily life, yet it remains notoriously difficult to capture and understand. Effective food journaling is thus a grand challenge in personal health informatics. In this paper we describe a system for food journaling called I Ate This, which is inspired by the Remote Food Photography Method (RFPM). I Ate This is simple: you use a smartphone app to take a photo and give a very basic description of any food or beverage you are about to consume. Later, a qualified dietitian will evaluate your photo, giving you feedback on how you did and where you can improve. The aim of I Ate This is to provide a convenient, visual and reliable way to help users learn from their eating habits and nudge them towards better choices each and every day. Ultimately, this incremental approach can lead to long-term behaviour change. Our goal is to bring RFPM to a wider audience, through APIs that can be incorporated into other apps.
BibTeX
@misc{goyal2017i,
  author = {Shubham Goyal and Qi Liu and Khairina Tajul-Arifin and Waqas Awan and Bimlesh Wadhwa and Zhenguang Liu},
  title = {{I Ate This: A Photo-based Food Journaling System with Expert Feedback.}},
  year = {2017},
  eprint = {1702.05957},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.05957},
}
Figure