TL;DR
This paper presents an automated pipeline for capturing the 3D shape and Bidirectional Texture Function (BTF) of food items. This allows for the creation of highly realistic, renderable 3D models of food that can be placed in arbitrary scenes with novel lighting, overcoming the significant manual effort typically required for such detailed digital content.
What Problem Does This Paper Address?
Creating convincing 3D models of everyday objects, especially food, is a significant challenge in computer graphics. Humans are highly attuned to the visual nuances of food, making even minor inconsistencies in shape, texture, or reflectance jarring. Traditional methods often involve extensive manual modeling by skilled 3D artists, which is time-consuming and expensive. Furthermore, for realistic rendering in arbitrary scenes, not only the geometry but also the spatially and angularly varying reflectance properties (BTF) need to be captured. This paper addresses the need for an automated, efficient, and accurate method to capture both the geometry and BTF of food items, specifically considering the constraints of food degradation and the need for fast acquisition.
Methods
The authors propose a parallelized system for automated capturing of geometry and Bidirectional Texture Function (BTF) of food. The core of their approach is an integrated capture system that benefits from extensive parallelization.
- Capturing System: A hemispherical gantry houses 151 consumer cameras and 8 projectors. This setup allows for dense acquisition of reflectance and geometry. The environment is actively cooled to mitigate food degradation during capture.
- Geometry Reconstruction: A structured light approach is employed, utilizing the 8 projectors. This method allows for capturing complex object geometries without moving the object and enables super-resolution for high-quality point clouds. The resulting correspondences are used for camera calibration via sparse bundle adjustment, and a triangle mesh is generated using Poisson Surface Reconstruction.
- Reflectance Reconstruction: The internal flashes of the cameras are used as light sources, triggered sequentially. All cameras simultaneously capture high dynamic range images. Standard reflectors are used for per-flash exposure calibration to ensure realistic color reproduction. By projecting images onto the acquired shape and combining reflectance values from different view- and light-directions, a BTF is estimated.
- Data Compression: The raw BTF data can be massive. The authors employ a Full-Matrix-Factorization technique for efficient compression, achieving a reduction of three orders of magnitude.
Datasets
Not discussed in the paper. The paper focuses on the capture methodology and its application to various food samples.
Key Insights and Ideas
- Automated Food Modeling is Achievable: The paper demonstrates that it's possible to automate the complex process of capturing realistic 3D food models, significantly reducing manual effort.
- Integrated Geometry and BTF Capture is Crucial: For realistic food rendering, capturing both detailed geometry and spatially/angularly varying reflectance (BTF) is essential, and their system achieves this simultaneously.
- Speed and Cooling are Paramount for Food: The sensitivity of food to degradation necessitates fast acquisition times and controlled environments (cooling) to maintain appearance during capture.
- BTF Captures Subtle Visual Cues: The BTF is shown to capture complex details like mesostructure, interreflections, local sub-surface scattering, and specularity, which are critical for the perceived realism of food.
- Compression is Key for Practical BTF Use: The significant data size of BTFs is addressed through effective compression techniques, making the data manageable for practical applications.
Key Results
The capture process for each food object took between 1.75 to 3.3 hours, depending on the dynamic range requirements. The output is a triangle mesh with a BTF of 1600x1600 pixels. Data compression reduces the dataset size from approximately 326 GB per object to about 1 GB, with further compression to 280MB possible with minimal impact on image quality. The rendered images (shown in Figures 1 and 3) exhibit a high degree of visual realism, accurately depicting highlights, fine surface details, and complex interactions like shadowing and indirect illumination.
Limitations and Future Work
- Data-Driven Reproduction: The technique is primarily suited for reproducing existing objects. While geometry can be modified, realistically adjusting texture and reflectance for novel shapes (e.g., adding seeds to a resized strawberry) requires BTF editing, which is an active research area.
- Degradation and Withering: Despite cooling and fast acquisition, some food items (like strawberry leaves) can still wither, leading to minor reprojection artifacts. Tracking changes over time could mitigate this.
- Sub-surface Scattering Limitations: While BTF captures local sub-surface scattering well, non-local sub-surface scattering effects (e.g., in grapes or cheese) are not fully handled. Future work could involve combining BTFs with models for global sub-surface scattering.
- Fine Structure Reconstruction: Poisson Surface Reconstruction can sometimes yield unsatisfactory results for very thin protrusions, potentially requiring manual correction. Improving the handling of such fine structures is a future direction.
Summary updated