A Novel Approach to Dining Bowl Reconstruction for Image-Based Food Volume Estimation

Wenyan Jia, Yiqiu Ren, Boyang Li, Britney Beatrice, Jingda Que, Shunxin Cao, Zekun Wu, Zhi‐Hong Mao, Benny Lo, Alex Kojo Anderson, Gary Frost, Megan A. McCrory, Edward Sazonov, Matilda Steiner‐Asiedu, Tom Baranowski, Lora E. Burke, Mingui SunSensors 2022 · DOI

TL;DR

This paper introduces a novel method for accurately estimating food volume in bowls using a single image. By applying a specially marked paper ruler to the bowl and analyzing its distortions in a photograph, the system reconstructs the bowl's 3D shape and then estimates food volume, outperforming human estimators.

What Problem Does This Paper Address?

Accurately tracking dietary intake is crucial for health and disease prevention. While image-based methods using smartphones or wearables are promising for dietary assessment, a significant challenge remains: computationally estimating the volume of food, especially in common containers like bowls. This is particularly difficult for amorphous foods where traditional methods struggle with 2D image limitations (lack of depth information, missing scale). Existing approaches often rely on expensive depth sensors or require manual intervention, limiting their practicality for widespread adoption. This paper tackles the problem of accurately and conveniently estimating food volume in round bowls from a single image, a scenario prevalent in many parts of the world.

Methods

The core of the proposed method lies in a clever way to "encode" the bowl's 3D shape into a 2D image.

  1. Bowl Reconstruction:
    • An adhesive paper ruler with evenly spaced markers is applied to the inside of the bowl, covering the bottom and sides.
  2. A single photograph of the taped bowl is taken from directly above.
  3. The key insight is that the distortions in the ruler's width and the spacing between markers in the image directly correspond to the bowl's 3D geometry.
  4. A computational algorithm analyzes these distortions to reconstruct the 3D interior surface of the bowl. This involves:
  5. Landmark Labeling: Identifying and labeling specific markers on the ruler in the image.
  6. Camera Modeling: Using a pinhole camera model and least-squares fitting to estimate the relationship between image points and 3D space, accounting for camera intrinsics.
  7. Parameter Estimation: An optimization process is used to find the optimal 3D distances and angles that best explain the observed ruler distortions, constrained by the assumption of a flat bowl bottom.
  8. Cross-Section Curve Reconstruction: The optimized parameters are used to reconstruct the bowl's cross-sectional curve, which is then rotated 360 degrees to form the complete 3D interior.
  9. Food Volume Estimation:
    • Once the bowl's 3D shape is reconstructed, its location and orientation in a new image (containing food) are estimated.
  10. The reconstructed 3D bowl is projected onto the 2D image.
  11. Pre-defined volumetric levels (e.g., 50 mL increments) are superimposed onto the projected bowl.
  12. The food volume is then estimated by determining how many of these levels the food occupies. For liquids with flat surfaces, this involves matching the liquid's surface area. For amorphous foods, a computer-assisted procedure with a sliding bar helps users visually match the food's level to the volumetric markers.

Datasets

  • Bowl Measurement: Nine commonly used bowls of varying shapes and sizes were used for ground truth measurements and algorithm validation.
  • Liquid Volume Estimation: 26 liquid samples in the nine bowls were tested. Images were captured using both a smartphone and an "eButton" wearable device.
  • Amorphous Food Volume Estimation: 114 real-world amorphous food samples across three different bowls were used. For each food, two images were taken under variable conditions.
  • Human vs. Algorithm Comparison: 228 images of amorphous foods were used to compare the algorithm's performance against a registered dietitian.

Key Insights and Ideas

  • Ruler Distortions as 3D Encoders: The paper ingeniously leverages the geometric distortions of a simple paper ruler to infer complex 3D bowl shapes. This is a low-cost, accessible way to capture crucial 3D information from a 2D image.
  • "Fullness" as a Superior Metric: The study suggests that "fullness" (food volume / bowl volume) is a more robust and intuitive metric for volume estimation than absolute units like milliliters, both for humans and algorithms. This simplifies the problem by focusing on relative comparisons within the container.
  • Computer Outperforms Human Estimators: The results demonstrate that the developed computational approach, using the reconstructed bowl as a volumetric reference, significantly outperforms human estimation, even by experienced dietitians. This highlights the potential of AI in objective dietary assessment.
  • Amorphous Foods are Still Challenging: While the method shows promise, estimating the exact "level" of amorphous foods, especially with natural peaks and valleys, remains a complex problem that may benefit from machine learning.

Key Results

  • Bowl Reconstruction Accuracy: Relative errors for estimated bowl volumes were generally below 5%, with a maximum error of 10.6%. The method showed robustness across different bowl shapes, though outward-curled rims presented a challenge.
  • Liquid Volume Estimation: For liquid samples, median fullness errors were between -8.7% and -2.8%, and median relative errors for volume were between -18.6% and -7.0%. The "fullness" metric generally showed better accuracy than direct volume estimation.
  • Amorphous Food Volume Estimation: Researchers using the software achieved median fullness errors below 5%, indicating good consistency.
  • Human vs. Algorithm Comparison: The computational tool consistently outperformed the registered dietitian in estimating food volumes, especially when using the "fullness" metric without additional visual cues. The dietitian's estimates improved with a visual cue (water bottle), but still lagged behind the algorithm.

Limitations and Future Work

  • Assumptions: The method relies on several assumptions, including a perfectly circular bowl rim, the ruler being taped centrally, and the camera being directly overhead. Deviations from these can introduce errors.
  • Automation: Currently, manual intervention is required for landmark labeling and determining the food level. The authors suggest that image processing algorithms could automate marker detection, but food level estimation for amorphous foods remains a significant challenge.
  • Amorphous Food Level Estimation: Identifying "peaks" and "valleys" in amorphous foods and accurately inferring a flattened level is difficult. The authors are exploring machine learning approaches to address this.
  • Non-Circular Bowls: The current algorithm is designed for round bowls and cannot handle irregularly shaped containers.
  • Camera Viewpoint and Rotation: The authors note that their algorithm could be improved by removing assumptions about camera viewpoint and rotation, but this would require a more complex mathematical model. They are actively researching these areas.

Summary updated

Abstract

Knowing the amounts of energy and nutrients in an individual's diet is important for maintaining health and preventing chronic diseases. As electronic and AI technologies advance rapidly, dietary assessment can now be performed using food images obtained from a smartphone or a wearable device. One of the challenges in this approach is to computationally measure the volume of food in a bowl from an image. This problem has not been studied systematically despite the bowl being the most utilized food container in many parts of the world, especially in Asia and Africa. In this paper, we present a new method to measure the size and shape of a bowl by adhering a paper ruler centrally across the bottom and sides of the bowl and then taking an image. When observed from the image, the distortions in the width of the paper ruler and the spacings between ruler markers completely encode the size and shape of the bowl. A computational algorithm is developed to reconstruct the three-dimensional bowl interior using the observed distortions. Our experiments using nine bowls, colored liquids, and amorphous foods demonstrate high accuracy of our method for food volume estimation involving round bowls as containers. A total of 228 images of amorphous foods were also used in a comparative experiment between our algorithm and an independent human estimator. The results showed that our algorithm overperformed the human estimator who utilized different types of reference information and two estimation methods, including direct volume estimation and indirect estimation through the fullness of the bowl.
BibTeX
@inproceedings{jia2022novel,
  author = {Wenyan Jia and Yiqiu Ren and Boyang Li and Britney Beatrice and Jingda Que and Shunxin Cao and Zekun Wu and Zhi‐Hong Mao and Benny Lo and Alex Kojo Anderson and Gary Frost and Megan A. McCrory and Edward Sazonov and Matilda Steiner‐Asiedu and Tom Baranowski and Lora E. Burke and Mingui Sun},
  title = {{A Novel Approach to Dining Bowl Reconstruction for Image-Based Food Volume Estimation}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22041493},
}
Figure