AI-enabled wearable cameras for assisting dietary assessment in African populations

Frank P.-w. Lo, Jianing Qiu, Modou L. Jobarteh, Yingnan Sun, Zeyu Wang, Shuo Jiang, Tom Baranowski, Alex K. Anderson, Megan A. McCrory, Edward Sazonov, Wenyan Jia, Mingui Sun, Matilda Steiner-Asiedu, Gary S. Frost, Benny LoNpj Digit. Medicine 2024 · DOI · OpenReview

TL;DR

This paper introduces EgoDiet, a novel AI-powered system using low-cost wearable cameras for passive dietary assessment. It significantly improves portion size estimation accuracy compared to traditional methods, offering a more practical and less burdensome solution, especially for low- and middle-income countries.

What Problem Does This Paper Address?

Accurate dietary assessment is crucial for public health, particularly in regions like Africa facing a "double burden of malnutrition." Traditional methods like 24-hour dietary recalls (24HR) are labor-intensive, expensive, prone to self-reporting bias, and can influence eating behavior. While mobile phone-based tools have emerged, they still require user interaction and technological literacy. This paper addresses the need for a passive, low-cost, and automated dietary assessment system that minimizes user burden and bias, making it suitable for large-scale studies in diverse populations.

Methods

EgoDiet is a multi-module pipeline:

  • EgoDiet:SegNet: Utilizes a Mask R-CNN architecture, pre-trained on MS COCO and fine-tuned on African food datasets, to segment food items and containers from egocentric camera views. This module is crucial for identifying what is being eaten and where.
  • EgoDiet:3DNet: A depth estimation network (encoder-decoder) that infers depth maps from RGB images to reconstruct 3D models of food containers. This allows for volume estimation without expensive depth sensors, addressing scale ambiguity.
  • EgoDiet:Feature: Extracts handcrafted features like Food Type (FT), Food Region Ratio (FRR), Volume of Container (VC), Plate Aspect Ratio (PAR), Pixel Number of the Plate (NP), and Area Weight Ratio (AWR). These features are designed to be informative for portion size estimation.
  • EgoDiet:PortionNet: Employs an ensemble machine learning approach (combining Random Forest, Extra Trees, and Multi-layer Perceptron) to estimate the weight of consumed food using the extracted features. This module is trained on weight-annotated data.

The system is designed for passive capture, meaning cameras continuously record eating episodes, and the AI pipeline processes this data offline.

Datasets

  • Diabetes60 dataset: Used in conjunction with self-constructed African food datasets for training the EgoDiet:SegNet module.
  • EgoDIMCAP-Lab dataset: A self-constructed dataset used for evaluating EgoDiet:3DNet's container volume estimation performance. This dataset includes paired RGB and depth images of containers with measured volumes.
  • African food dataset: A self-constructed dataset used for training the portion size estimation module (EgoDiet:PortionNet).

Key Insights and Ideas

  • Passive monitoring is feasible and effective: The study demonstrates that low-cost wearable cameras can capture sufficient data for accurate dietary assessment without active user intervention.
  • Feature engineering is key for limited data: By leveraging domain knowledge to extract specific features (like FRR), the system can achieve good performance even with limited weight-annotated data, overcoming a major bottleneck in portion size estimation.
  • Container volume estimation without depth sensors: EgoDiet:3DNet shows that standard RGB cameras can be used to estimate container volume, making the system more accessible and cost-effective.
  • Ensemble learning for robust portion size estimation: Combining multiple ML models in an ensemble approach leads to improved accuracy and lower error rates in estimating food weight.
  • The importance of Food Region Ratio (FRR): Permutation importance analysis revealed FRR as the most significant feature for portion size estimation, highlighting the value of accurate food segmentation.

Key Results

  • Study A (London): EgoDiet achieved a Mean Absolute Percentage Error (MAPE) of 31.9% for portion size estimation, outperforming dietitians' visual estimates (MAPE of 40.1%).
  • Study B (Ghana): EgoDiet demonstrated a MAPE of 28.0%, showing a reduction in error compared to the traditional 24-Hour Dietary Recall (24HR) method (MAPE of 32.5%).
  • EgoDiet:3DNet: Achieved a Mean Absolute Error (MAE) comparable to depth camera measurements for container volume estimation.
  • Ensemble ML model: With all handcrafted features, the ensemble model achieved the best performance with an MAE of 33.8g for portion size estimation.

Limitations and Future Work

  • Privacy concerns: Passive monitoring raises privacy issues, which the authors address through AI-driven face blurring and data encryption, but acknowledge room for optimization.
  • Accuracy compared to active methods: While promising, EgoDiet's accuracy in portion size estimation is still slightly lower than some active capturing methods, partly due to challenges with consistent camera positioning and hand occlusion.
  • Fine-grained food recognition: The current model focuses on broader food categories; future work aims to achieve more detailed and specific food item recognition.
  • Handling complex eating scenarios: Challenges remain in accurately assessing food from shared plates and distinguishing between eating and food preparation.
  • Hardware improvements: Future work will focus on enhancing camera resolution, field of view, and ensuring consistent capture angles.
  • Integration of foundational models: Leveraging large-scale foundational models for improved food recognition and contextual understanding is a promising direction.
  • Action recognition: Incorporating learning-based action recognition will help differentiate various daily activities, further enhancing the system's intelligence.

Summary updated

Abstract

We have developed a population-level method for dietary assessment using low-cost wearable cameras. Our approach, EgoDiet, employs an egocentric vision-based pipeline to learn portion sizes, addressing the shortcomings of traditional self-reported dietary methods. To evaluate the functionality of this method, field studies were conducted in London (Study A) and Ghana (Study B) among populations of Ghanaian and Kenyan origin. In Study A, EgoDiet’s estimations were contrasted with dietitians’ assessments, revealing a performance with a Mean Absolute Percentage Error (MAPE) of 31.9% for portion size estimation, compared to 40.1% for estimates made by dietitians. We further evaluated our approach in Study B, comparing its performance to the traditional 24-Hour Dietary Recall (24HR). Our approach demonstrated a MAPE of 28.0%, showing a reduction in error when contrasted with the 24HR, which exhibited a MAPE of 32.5%. This improvement highlights the potential of using passive camera technology to serve as an alternative to the traditional dietary assessment methods.
BibTeX
@article{lo2024ai,
  author = {Frank P.-W. Lo and Jianing Qiu and Modou L. Jobarteh and Yingnan Sun and Zeyu Wang and Shuo Jiang and Tom Baranowski and Alex K. Anderson and Megan A. McCrory and Edward Sazonov and Wenyan Jia and Mingui Sun and Matilda Steiner-Asiedu and Gary S. Frost and Benny Lo},
  title = {{AI-enabled wearable cameras for assisting dietary assessment in African populations}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-024-01346-8},
}
Figure