TL;DR
This paper introduces the MetaFood CVPR 2024 Challenge, a competition focused on physically informed 3D food reconstruction for accurate dietary monitoring. The challenge highlights the need for volumetric accuracy beyond just surface geometry, and the winning teams demonstrate novel approaches to reconstruct food items from limited 2D inputs, achieving significant improvements in volume estimation error.
What Problem Does This Paper Address?
Accurately estimating food portion sizes is crucial for dietary assessment and nutritional monitoring. Traditional methods often rely on manual input or 2D images, which lack the necessary 3D information for precise volume estimation. While 3D reconstruction techniques have advanced, they often focus on surface geometry and struggle with the unique challenges of food items (varying textures, reflective surfaces, complex shapes) and the practical constraints of real-world use (limited views, scarce data). This paper addresses this gap by hosting a challenge and presenting winning solutions that aim to reconstruct food items with both accurate shape and volume from limited 2D data, using physical references for scaling.
Methods
The MetaFood challenge itself focuses on reconstructing 3D models of food items from 2D images, using a visible checkerboard as a size reference. The challenge was structured into three difficulty levels: easy (200 images), medium (30 images), and hard (1 image).
The winning teams employed various sophisticated techniques:
- VolETA (First Place): This team utilized a few-shot approach combining computer vision and deep learning. They focused on keyframe selection to ensure data quality, employed PixSfM for camera pose estimation, and used SAM and XMem++ for object segmentation. Their core reconstruction was done using NeuS2, followed by a "Remove Isolated Pieces" technique for mesh refinement. A crucial step involved manually identifying and then fine-tuning a scaling factor using depth information and object masks to achieve real-world scaling.
- ININ-VIAUN (Second Place): This team's pipeline involved a corner projection matching method for scale factor estimation, leveraging COLMAP for dense point clouds and image poses. For 3D reconstruction, they used a combination of COLMAP, DiffusioNeRF, and NeRF2Mesh for multi-view inputs, and ZeroNVS for single-view inputs. Mesh refinement was performed using MeshFix for holes and Laplacian Smoothing for surface noise.
- FoodRiddle (Best 3D Mesh Reconstruction): This team designed two pipelines: a structure-from-motion (SfM) approach for multi-view inputs (extending COLMAP with SuperPoint and SuperGlue) and a single-view reconstruction pipeline using an AIGC method to generate multi-view images from a single input. Both pipelines involved mesh reconstruction, scale adjustment using the checkerboard, and post-processing for refinement.
Datasets
The challenge utilized the MetaFood3D dataset, comprising 20 carefully selected food items. Each item was scanned with a 3D scanner and captured in video sequences. To ensure accurate scaling, the captures included a checkerboard and pattern mat. The dataset was organized into three difficulty levels based on the number of available 2D images for reconstruction:
- Easy: ~200 images
- Medium: 30 images
- Hard: 1 image
Key Insights and Ideas
- The Importance of Physical References: The explicit inclusion of a checkerboard as a physical reference for scaling was critical. This allowed teams to move beyond relative scale and achieve real-world, volumetric accuracy, which is paramount for portion estimation.
- Few-Shot and Single-View Reconstruction are Viable: The success of teams like VolETA and ININ-VIAUN in achieving good results with limited image inputs (especially the "hard" category with a single image) demonstrates the growing power of few-shot and single-view 3D reconstruction techniques in practical scenarios.
- Mesh Refinement is Crucial: Techniques like "Remove Isolated Pieces," MeshFix, and Laplacian Smoothing were essential for producing clean, accurate, and usable 3D meshes, addressing issues like holes and noise.
- Complementary Strengths: The different approaches of the winning teams highlight that there isn't a single "best" method for all food items and conditions. Some methods excel in volume accuracy, while others are better at detailed mesh reconstruction.
Key Results
- Overall Performance: The challenge saw promising results in 3D food reconstruction, with significant potential for improving portion estimation.
- VolETA (First Place): Achieved the overall best performance in both volume accuracy (Phase-I) and shape accuracy (Phase-II).
- FoodRiddle (Best 3D Mesh Reconstruction): Demonstrated superior performance in Phase-II, indicating a high-caliber field for 3D mesh reconstruction.
- Quantitative Improvements: The paper mentions that the three winning methods achieved 18-23% lower volume estimation error compared to the current state-of-the-art models.
- Qualitative Results: The qualitative results showed good texture details, artifact correction, and handling of missing data across different food items.
Limitations and Future Work
- Manual Processes: Some teams acknowledged manual steps in their pipelines, such as providing segmentation prompts or identifying scaling factors. Automating these processes would enhance efficiency and applicability.
- Input Requirements: The methods often require extensive input information (e.g., food masks, depth data), and streamlining these requirements would simplify the process.
- Complex Backgrounds and Objects: The current methods have not been extensively tested on highly complex backgrounds or intricate food objects, suggesting a need for further validation in more challenging scenarios.
- Capturing Complexities: The robustness of the methods under varying capturing conditions (distance, camera speed) needs further investigation.
- Pipeline Complexity: For single-view reconstruction, some teams aim to simplify their pipelines by exploring more efficient models like diffusion models.
Summary updated