TL;DR
This paper introduces a novel method for accurately estimating food volume in bowls using a single image. By applying a specially marked paper ruler to the bowl and analyzing its distortions in a photograph, the system reconstructs the bowl's 3D shape and then estimates food volume, outperforming human estimators.
What Problem Does This Paper Address?
Accurately tracking dietary intake is crucial for health and disease prevention. While image-based methods using smartphones or wearables are promising for dietary assessment, a significant challenge remains: computationally estimating the volume of food, especially in common containers like bowls. This is particularly difficult for amorphous foods where traditional methods struggle with 2D image limitations (lack of depth information, missing scale). Existing approaches often rely on expensive depth sensors or require manual intervention, limiting their practicality for widespread adoption. This paper tackles the problem of accurately and conveniently estimating food volume in round bowls from a single image, a scenario prevalent in many parts of the world.
Methods
The core of the proposed method lies in a clever way to "encode" the bowl's 3D shape into a 2D image.
- Bowl Reconstruction:
- An adhesive paper ruler with evenly spaced markers is applied to the inside of the bowl, covering the bottom and sides.
- A single photograph of the taped bowl is taken from directly above.
- The key insight is that the distortions in the ruler's width and the spacing between markers in the image directly correspond to the bowl's 3D geometry.
- A computational algorithm analyzes these distortions to reconstruct the 3D interior surface of the bowl. This involves:
- Landmark Labeling: Identifying and labeling specific markers on the ruler in the image.
- Camera Modeling: Using a pinhole camera model and least-squares fitting to estimate the relationship between image points and 3D space, accounting for camera intrinsics.
- Parameter Estimation: An optimization process is used to find the optimal 3D distances and angles that best explain the observed ruler distortions, constrained by the assumption of a flat bowl bottom.
- Cross-Section Curve Reconstruction: The optimized parameters are used to reconstruct the bowl's cross-sectional curve, which is then rotated 360 degrees to form the complete 3D interior.
- Food Volume Estimation:
- Once the bowl's 3D shape is reconstructed, its location and orientation in a new image (containing food) are estimated.
- The reconstructed 3D bowl is projected onto the 2D image.
- Pre-defined volumetric levels (e.g., 50 mL increments) are superimposed onto the projected bowl.
- The food volume is then estimated by determining how many of these levels the food occupies. For liquids with flat surfaces, this involves matching the liquid's surface area. For amorphous foods, a computer-assisted procedure with a sliding bar helps users visually match the food's level to the volumetric markers.
Datasets
- Bowl Measurement: Nine commonly used bowls of varying shapes and sizes were used for ground truth measurements and algorithm validation.
- Liquid Volume Estimation: 26 liquid samples in the nine bowls were tested. Images were captured using both a smartphone and an "eButton" wearable device.
- Amorphous Food Volume Estimation: 114 real-world amorphous food samples across three different bowls were used. For each food, two images were taken under variable conditions.
- Human vs. Algorithm Comparison: 228 images of amorphous foods were used to compare the algorithm's performance against a registered dietitian.
Key Insights and Ideas
- Ruler Distortions as 3D Encoders: The paper ingeniously leverages the geometric distortions of a simple paper ruler to infer complex 3D bowl shapes. This is a low-cost, accessible way to capture crucial 3D information from a 2D image.
- "Fullness" as a Superior Metric: The study suggests that "fullness" (food volume / bowl volume) is a more robust and intuitive metric for volume estimation than absolute units like milliliters, both for humans and algorithms. This simplifies the problem by focusing on relative comparisons within the container.
- Computer Outperforms Human Estimators: The results demonstrate that the developed computational approach, using the reconstructed bowl as a volumetric reference, significantly outperforms human estimation, even by experienced dietitians. This highlights the potential of AI in objective dietary assessment.
- Amorphous Foods are Still Challenging: While the method shows promise, estimating the exact "level" of amorphous foods, especially with natural peaks and valleys, remains a complex problem that may benefit from machine learning.
Key Results
- Bowl Reconstruction Accuracy: Relative errors for estimated bowl volumes were generally below 5%, with a maximum error of 10.6%. The method showed robustness across different bowl shapes, though outward-curled rims presented a challenge.
- Liquid Volume Estimation: For liquid samples, median fullness errors were between -8.7% and -2.8%, and median relative errors for volume were between -18.6% and -7.0%. The "fullness" metric generally showed better accuracy than direct volume estimation.
- Amorphous Food Volume Estimation: Researchers using the software achieved median fullness errors below 5%, indicating good consistency.
- Human vs. Algorithm Comparison: The computational tool consistently outperformed the registered dietitian in estimating food volumes, especially when using the "fullness" metric without additional visual cues. The dietitian's estimates improved with a visual cue (water bottle), but still lagged behind the algorithm.
Limitations and Future Work
- Assumptions: The method relies on several assumptions, including a perfectly circular bowl rim, the ruler being taped centrally, and the camera being directly overhead. Deviations from these can introduce errors.
- Automation: Currently, manual intervention is required for landmark labeling and determining the food level. The authors suggest that image processing algorithms could automate marker detection, but food level estimation for amorphous foods remains a significant challenge.
- Amorphous Food Level Estimation: Identifying "peaks" and "valleys" in amorphous foods and accurately inferring a flattened level is difficult. The authors are exploring machine learning approaches to address this.
- Non-Circular Bowls: The current algorithm is designed for round bowls and cannot handle irregularly shaped containers.
- Camera Viewpoint and Rotation: The authors note that their algorithm could be improved by removing assumptions about camera viewpoint and rotation, but this would require a more complex mathematical model. They are actively researching these areas.
Summary updated