Physically Informed 3D Food Reconstruction: Methods and Results

Jiangpeng He, Yuhao Chen, Gautham Vinod, Xiaoyan Zhang, Talha Ibn Mahmud, Ahmad AlMughrabi, Umair Haroon, Ricardo Marques, Petia Radeva, Yawei Jueluo, Chengyu Shi, Pengyu Wang, Pengcheng Xi, Alex Wong, Fengqing ZhuIEEE JBHI 2026 · DOI

TL;DR

This paper introduces the MetaFood CVPR 2024 Challenge, a competition focused on physically informed 3D food reconstruction for accurate dietary monitoring. The challenge highlights the need for volumetric accuracy beyond just surface geometry, and the winning teams demonstrate novel approaches to reconstruct food items from limited 2D inputs, achieving significant improvements in volume estimation error.

What Problem Does This Paper Address?

Accurately estimating food portion sizes is crucial for dietary assessment and nutritional monitoring. Traditional methods often rely on manual input or 2D images, which lack the necessary 3D information for precise volume estimation. While 3D reconstruction techniques have advanced, they often focus on surface geometry and struggle with the unique challenges of food items (varying textures, reflective surfaces, complex shapes) and the practical constraints of real-world use (limited views, scarce data). This paper addresses this gap by hosting a challenge and presenting winning solutions that aim to reconstruct food items with both accurate shape and volume from limited 2D data, using physical references for scaling.

Methods

The MetaFood challenge itself focuses on reconstructing 3D models of food items from 2D images, using a visible checkerboard as a size reference. The challenge was structured into three difficulty levels: easy (200 images), medium (30 images), and hard (1 image).

The winning teams employed various sophisticated techniques:

  • VolETA (First Place): This team utilized a few-shot approach combining computer vision and deep learning. They focused on keyframe selection to ensure data quality, employed PixSfM for camera pose estimation, and used SAM and XMem++ for object segmentation. Their core reconstruction was done using NeuS2, followed by a "Remove Isolated Pieces" technique for mesh refinement. A crucial step involved manually identifying and then fine-tuning a scaling factor using depth information and object masks to achieve real-world scaling.
  • ININ-VIAUN (Second Place): This team's pipeline involved a corner projection matching method for scale factor estimation, leveraging COLMAP for dense point clouds and image poses. For 3D reconstruction, they used a combination of COLMAP, DiffusioNeRF, and NeRF2Mesh for multi-view inputs, and ZeroNVS for single-view inputs. Mesh refinement was performed using MeshFix for holes and Laplacian Smoothing for surface noise.
  • FoodRiddle (Best 3D Mesh Reconstruction): This team designed two pipelines: a structure-from-motion (SfM) approach for multi-view inputs (extending COLMAP with SuperPoint and SuperGlue) and a single-view reconstruction pipeline using an AIGC method to generate multi-view images from a single input. Both pipelines involved mesh reconstruction, scale adjustment using the checkerboard, and post-processing for refinement.

Datasets

The challenge utilized the MetaFood3D dataset, comprising 20 carefully selected food items. Each item was scanned with a 3D scanner and captured in video sequences. To ensure accurate scaling, the captures included a checkerboard and pattern mat. The dataset was organized into three difficulty levels based on the number of available 2D images for reconstruction:

  • Easy: ~200 images
  • Medium: 30 images
  • Hard: 1 image

Key Insights and Ideas

  • The Importance of Physical References: The explicit inclusion of a checkerboard as a physical reference for scaling was critical. This allowed teams to move beyond relative scale and achieve real-world, volumetric accuracy, which is paramount for portion estimation.
  • Few-Shot and Single-View Reconstruction are Viable: The success of teams like VolETA and ININ-VIAUN in achieving good results with limited image inputs (especially the "hard" category with a single image) demonstrates the growing power of few-shot and single-view 3D reconstruction techniques in practical scenarios.
  • Mesh Refinement is Crucial: Techniques like "Remove Isolated Pieces," MeshFix, and Laplacian Smoothing were essential for producing clean, accurate, and usable 3D meshes, addressing issues like holes and noise.
  • Complementary Strengths: The different approaches of the winning teams highlight that there isn't a single "best" method for all food items and conditions. Some methods excel in volume accuracy, while others are better at detailed mesh reconstruction.

Key Results

  • Overall Performance: The challenge saw promising results in 3D food reconstruction, with significant potential for improving portion estimation.
  • VolETA (First Place): Achieved the overall best performance in both volume accuracy (Phase-I) and shape accuracy (Phase-II).
  • FoodRiddle (Best 3D Mesh Reconstruction): Demonstrated superior performance in Phase-II, indicating a high-caliber field for 3D mesh reconstruction.
  • Quantitative Improvements: The paper mentions that the three winning methods achieved 18-23% lower volume estimation error compared to the current state-of-the-art models.
  • Qualitative Results: The qualitative results showed good texture details, artifact correction, and handling of missing data across different food items.

Limitations and Future Work

  • Manual Processes: Some teams acknowledged manual steps in their pipelines, such as providing segmentation prompts or identifying scaling factors. Automating these processes would enhance efficiency and applicability.
  • Input Requirements: The methods often require extensive input information (e.g., food masks, depth data), and streamlining these requirements would simplify the process.
  • Complex Backgrounds and Objects: The current methods have not been extensively tested on highly complex backgrounds or intricate food objects, suggesting a need for further validation in more challenging scenarios.
  • Capturing Complexities: The robustness of the methods under varying capturing conditions (distance, camera speed) needs further investigation.
  • Pipeline Complexity: For single-view reconstruction, some teams aim to simplify their pipelines by exploring more efficient models like diffusion models.

Summary updated

Abstract

Accurate food portion size estimation is a critical challenge in nutrition analysis and dietary assessment. Recent 3D reconstruction methods primarily focus on surface geometry, often neglecting the volumetric accuracy necessary for precise portion size estimation. In this work, we present three methods, NS-DRS, HR-CMS, and GS-GP, for real-scale 3D food reconstruction and volume estimation from limited 2D inputs. All three methods follow a shared three-stage pipeline consisting of 3D reconstruction, scale estimation using physical references such as visible checkerboards, and mesh refinement. While this high-level structure is consistent, the methods differ in their reconstruction backbones and scale-recovery mechanisms, and are designed for different input conditions, including both single-view and multi-view images. We evaluate each method on a diverse set of food items exhibiting variations in texture, shape, and camera pose, and assess their performance in terms of volume estimation and geometric accuracy. The results reveal complementary strengths among the pipelines, with NS-DRS performing better in volume estimation and GS-GP being more effective in 3D reconstruction. Across all settings, the three methods achieve 18-23% lower volume estimation error compared to the current state-of-the-art model. These findings demonstrate the effectiveness of physically informed and explainable reconstruction pipelines for accurate portion estimation, and support their potential use as scalable tools for dietary monitoring and clinical nutrition analysis.
BibTeX
@article{he2026physically,
  author = {Jiangpeng He and Yuhao Chen and Gautham Vinod and Xiaoyan Zhang and Talha Ibn Mahmud and Ahmad AlMughrabi and Umair Haroon and Ricardo Marques and Petia Radeva and Yawei Jueluo and Chengyu Shi and Pengyu Wang and Pengcheng Xi and Alex Wong and Fengqing Zhu},
  title = {{Physically Informed 3D Food Reconstruction: Methods and Results}},
  journal = {IEEE Journal of Biomedical and Health Informatics},
  year = {2026},
  doi = {10.1109/jbhi.2026.3669573},
}
Figure