Abstract
Automating dietary assessment is a critical goal for public health, yet accurate volume estimation from 2D images remains a significant challenge due to the lack of large-scale, 3D-labeled datasets. This paper presents a novel GAN-based architecture designed to generate realistic food images that strictly preserve the original shape and volume of the food and its container. By decoupling shape, texture, and category, the proposed method enables "swapping" food types within a fixed geometry. This approach provides a scalable solution for data augmentation in training AI models for dietary assessment and volume estimation.
Introduction
Traditional self-reporting of dietary intake is often inaccurate and biased. While modern AI can recognize food types with high precision, estimating the actual volume of food consumed from a single 2D image is much harder. Training models for volume estimation requires massive datasets with precise 3D labels, which are costly and difficult to acquire.
Generative Adversarial Networks (GANs) offer a potential solution through data augmentation. However, standard GANs frequently distort object shapes or "hallucinate" features, rendering the synthetic data useless for volume-related tasks. If the shape of the food changes during generation, the underlying volume information is lost. This research addresses this gap by introducing a shape-preserving conditional food image generation framework.
Proposed Method
The core of the proposed system is a GAN-based neural network architecture that treats food image generation as a constrained translation task. The architecture consists of three primary modules:
- Shape Encoder ($E$): Unlike typical GANs that initialize from noise, this model employs an encoder to extract essential topological features from a reference "shape image." It focuses on "low-frequency" information—the general outline and structure—using specific downsampling layers.
- Generator ($G$): The generator synthesizes the final image by combining three inputs: shape features ($f$) from the encoder, a latent variable ($z$) for fine-grained texture control, and a category label ($c$) to specify the food type.
- Discriminator ($D$): A conditional discriminator evaluates the realism of the generated image relative to the specified food category.
The training process utilizes a dual-loss strategy. In addition to the standard GAN loss for realism, a Reconstruction loss ($L_1$) is applied to ensure that the encoder can still identify the original shape features in the generated output, thereby enforcing strict geometric fidelity.
Datasets and Evaluation
The model was trained and evaluated using three major datasets:
- VireoFood-172: A large-scale Chinese food dataset (172 categories, ~110k images).
- Food-101: A benchmark Western food dataset (~101k images).
- UEC-FoodPIX: Specifically used for shape preservation evaluation due to its manual segmentation masks.
Results and Discussion
The proposed method demonstrates a remarkable ability to decouple geometry from semantics. By keeping the "shape" constant while varying the "category," the model can generate infinite variations of a meal with the exact same volume footprint (e.g., replacing rice with noodles on the same plate).
Quantitative Performance
The model achieved state-of-the-art results across several benchmarks:
- Realism: On VireoFood-172, it achieved a Fréchet Inception Distance (FID) of 4.97, significantly better than StyleGAN3's 9.25. On Food-101, it scored 22.82, outperforming StyleGAN3 (39.05) and Latent Diffusion (30.39).
- Shape Fidelity: Using the Intersection Over Union (IoU) metric, the model achieved an average score of 0.91, indicating that generated food almost perfectly adheres to the reference boundaries.
Limitations
Despite its success, the model faces challenges with "common sense" failures. For instance, it may attempt to generate liquid-based foods (like porridge) in the shape of a flat plate if the reference image was a plate. Additionally, intricate patterns on plates are sometimes misinterpreted as food, leading to "bleeding" effects where the generated food overlaps with the container's decoration.
Conclusion
This research provides a compact and effective solution for generating shape-constrained food images. By ensuring that synthetic data maintains the geometric properties of real-world references, this method paves the way for more robust AI-driven dietary assessment systems capable of accurate volume estimation. Future work will focus on refining container-food compatibility and improving the handling of complex background patterns.
Summary updated