Photo-based diet tracking fails most often on what the camera cannot see: oil under a salad, sauce beneath rice, a meat layer buried in a casserole. Current systems either ignore these components or guess them from which visible foods tend to co-occur with them. This proposal trains a model entirely on simulated dishes, where every buried layer is known exactly, to predict a stack of per-ingredient height layers from one overhead image of a real plate. The stack covers both what lies on top and what lies beneath. Multiplying each layer's volume by that ingredient's density gives per-ingredient mass, and from mass, calories.
The bet is about which part of a simulator can be trusted. Rendered food looks fake, and a model that learns appearance shortcuts in simulation tends to collapse on real photos. The geometry of how food piles, spreads and covers other food on a plate is much easier to simulate faithfully, and a depth sensor sees it with a smaller reality gap than color does. So simulation is used only to teach amodal geometry, meaning the shape of things including their hidden parts. Appearance and density are left to real data.
Core Claim
A monocular RGB-D model trained on procedurally simulated layered dishes, with depth and coarse ingredient labels as its main inputs, can recover the volume of occluded ingredients on real plates. It should recover them more accurately than co-occurrence inference, which estimates a hidden ingredient's amount from the visible ones. It should also beat end-to-end mass regression. The gain should be concentrated on ingredients that are covered by other food, and small on fully visible ones. That concentration is the signature that would distinguish a real amodal effect from a generic data-augmentation effect.
The mass of ingredient $k$ is read off the predicted stack as
$$m_k = \rho_k \sum_{x} \big(h_k(x) - h_{k-1}(x)\big)\,a(x),$$
where $h_k(x)$ is the predicted height at pixel $x$ of the surface after ingredient $k$ is placed, $h_{k-1}(x)$ is the surface just beneath it, $a(x)$ is the metric area of pixel $x$ obtained from depth, and $\rho_k$ is the ingredient's density.
Why This, and What Exists
Within dietary assessment, three papers motivate the gap. Nutrition5k built a robotic rig with a depth sensor and scales that records dishes as they are assembled ingredient by ingredient. It showed that depth cuts calorie error from 26.1% to 16.5%. It also notes that invisible components can only be inferred statistically. PITA predicts relative ingredient amounts from a shared image–recipe embedding and inherits the same co-occurrence limit.
The geometric methods constrain our design choices. Food Portion Estimation via 3D Object Scaling and VolTex obtain strong volume error, 14.01% and 4.46% MAPE respectively, by scaling CAD models or reconstructing surfaces. Both reconstruct only the visible outer surface of each item, so they say nothing about what lies underneath it.
The sim-to-real literature supplies the method and also warns where it breaks. MaviGen and the EUREKA model from Amodal Instance Segmentation with IRAIS procedurally render cluttered scenes with exact hidden-region labels. A model trained purely on these renders reached 37.10% AP on real retail shelves, but the authors report that it breaks down on deformable and transparent objects, which is exactly what food is. We reuse their recipe of procedural scenes with automatic amodal labels. We answer their failure mode by simulating deformable piling physically rather than placing rigid scans.
Driving in the Matrix shows that free depth labels from a rendering engine plus scene diversity can beat narrow real datasets. Sim-to-Seg shows that predicting a canonical intermediate, such as segmentation and depth, rather than the end task transfers better than raw domain randomization. Our height-layer stack is that kind of canonical intermediate, tailored to food.
PixelDA concedes that pixel-level translation fixes color and lighting but not 3D structure. This is why we do not rely on image translation to close the gap.
The closest food-and-synthetic work is DiSC-DS. It generates diffusion images to improve long-tailed food classification and explicitly leaves portion and calorie estimation open. Its generated images carry class labels only, with no geometry. Robotic slicing in simulation (SliceIt!) and synthetic olive detection (Olive the Above) touch food but not intake estimation.
Research Program
The simulator. We would build a procedural plate simulator with three ingredient families:
- Granular foods such as rice, peas and diced vegetables, piled with a particle physics simulator.
- Soft solids such as meat slices, fish, bread and leaves, as deformable meshes dropped and settled under gravity.
- Liquids and coatings such as sauces, dressings and oil, as thin films that flow into the low regions of whatever lies beneath.
Dishes are assembled in sampled orders that mimic real composition. Base starches come first, proteins next, toppings and dressings last. At each placement the engine records the full height field, giving exact ground-truth layers $h_k$ for free in the way game-engine depth buffers do.
The renderer outputs RGB and depth with a depth-sensor noise model covering holes at specular sauces and edge noise. Textures are randomized, including crops taken from real food photos, so that the model cannot use appearance as a shortcut.
The model. The model takes RGB-D plus an ingredient list and predicts the ordered stack. The ingredient list could come from a detector or from a recipe or diary text, which is the multimodal link to text pipelines like KitchenScale.
Densities. Densities $\rho_k$ are not simulated. They are fitted on real data, either from per-ingredient weights against measured volumes, or by a small learned head calibrated on the real training split.
Real evaluation of hidden layers. Nutrition5k's incremental assembly is what makes this testable on real food. The rig recorded the dish as ingredients were added, so the dish before an ingredient was added shows the true surface that the ingredient later hides. This assumes the per-step overhead captures are available as the dataset description suggests. If only per-step weights are usable, we evaluate per-ingredient mass and use the assembly order to label which ingredients were later covered.
Baselines
Naive reference. The do-nothing reference point uses only the visible surface: the top surface minus a fitted plate plane. That volume is split among detected ingredients by visible area and converted with the same densities. This measures how much error hidden ingredients actually cause.
Strongest existing method. The strongest competitor is the depth-aware multi-task regressor of Nutrition5k. It reports 16.5% calorie error and 26.2% aggregate macronutrient MAE. It is strongest because it already uses depth on the same data and can learn hidden-ingredient statistics implicitly. We would retrain it with the same backbone family, image resolution, training split, and hyperparameter search budget as our model. We would report it both at the dish level and, through a per-ingredient head, at the ingredient level. Beating it means lower dish calorie error than 16.5% on the same split. More importantly, it means a statistically significant reduction in mass error on the subset of ingredients labeled as covered.
Co-occurrence inference. The statistical approach the dietary papers describe gets its own baseline. A model predicts each hidden ingredient's mass from visible ingredients and the dish's visible volume, trained on real Nutrition5k data. This is the direct competitor on the hidden-ingredient question.
Real-only amodal supervision. We also train the same layer-stack architecture only on real Nutrition5k incremental layers. This is the most dangerous comparison and is discussed below.
Ablations. Two ablations isolate the key ingredient:
- A simulator variant that renders the same dishes but supervises only the top surface. If this matches the full model, layered supervision is not what helps.
- A variant replacing physical piling with random smooth height blobs. This tests whether physical plausibility of the geometry matters.
Why not just…?
Why not just train an RGB or RGB-D regressor end to end on Nutrition5k? That is the strongest baseline above, and it may well match us on dish-level calories, because hidden ingredients are a minority of dish mass in cafeteria plates. The proposal does not claim to win on that aggregate. It claims to win on covered ingredients, which the per-ingredient breakdown measures directly. The evidence that end-to-end regression is brittle is mixed. It does well inside the Nutrition5k rig, while 3D Object Scaling reports above 200% error for RGB regression on SimpleFood45. We would therefore also test cross-dataset transfer.
Why not just supervise the layer stack with Nutrition5k's real incremental scans and skip simulation? This is a real threat. If 5,000 real dishes suffice, simulation adds nothing. We settle it in two ways. The first is data-efficiency curves at 10%, 30% and 100% of real dishes, each with and without simulated pretraining. The second is a held-out split by ingredient and dish type, testing whether simulated diversity helps beyond the cafeteria-skewed real set. If real-only matches simulation-pretrained training at full data and on held-out types, we would report that simulation is useful only in low-data regimes, and the claim narrows accordingly.
Why not just use a checkerboard and surface reconstruction like VolTex? Surface reconstruction achieves excellent visible volume, 4.46% MAPE on MetaFood3D. However, a watertight outer mesh of a covered dish still assigns zero volume to the buried sauce, and it requires a reference board plus a video sweep. On SimpleFood45, where foods are single and uncovered, we would expect no advantage and would report that honestly. That benchmark serves as a check that our visible-volume estimates are not worse.
Why not just generate more training images with diffusion, as DiSC-DS does? Diffusion images arrive with no hidden-layer ground truth, so they cannot supervise the quantity this proposal targets. DiSC-DS itself needed a specialized sampler to avoid semantic collapse, even for class labels. We would still test diffusion-augmented training of the Nutrition5k regressor as a control. A matching gain on covered ingredients would mean the effect is generic augmentation rather than amodal geometry.
Proposed Experiments
Real hidden-ingredient test. The primary test runs on Nutrition5k with a dish-level split. Metrics are:
- per-ingredient mass MAE as a percentage, reported separately for covered and visible ingredients;
- dish-level calorie and macronutrient error, compared to the reported 16.5% and 26.2%;
- height-layer error against real incremental surfaces, where those surfaces are available.
The decisive check has two parts. First, the full simulation-pretrained model must reduce covered-ingredient mass error relative to both the co-occurrence baseline and the top-surface-only ablation. Second, the gain on visible ingredients must be small. If the gain is uniform across covered and visible ingredients, we cannot attribute it to amodal reasoning.
Sim-to-real diagnostics. We run zero-shot transfer from simulation only, then fine-tuning with increasing real data, then depth-only versus RGB-D input. Depth-only input should transfer better zero-shot if our premise about where the gap lives is correct. If it does not, the premise is wrong.
Transfer sanity checks. On SimpleFood45, which provides a checkerboard for scale, we report volume and energy MAPE against the reported 14.01% and 17.67%. On MetaFood3D we report volume MAPE against VolTex's 4.46%. Here we expect parity at best, and the purpose is to show that layer modeling does not degrade visible geometry.
Mixed dishes. A stew-like subset of Nutrition5k tests the regime where IRAIS-style amodal transfer is known to fail. This tells us whether physical piling and film simulation extend amodal transfer to deformable food, or whether well-mixed dishes remain out of reach.
References
- Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food
- Picture-to-Amount (PITA): Predicting Relative Ingredient Amounts from Food Images
- Food Portion Estimation via 3D Object Scaling
- VolTex: Food Volume Estimation using Text-Guided Segmentation and Neural Surface Reconstruction
- Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Synthetic Data Augmentation using Pre-trained Diffusion Models for Long-tailed Food Image Classification
- SliceIt!: Simulation-Based Reinforcement Learning for Compliant Robotic Food Slicing
- Detecting Olives with Synthetic or Real Data? Olive the Above
- KitchenScale: Learning to predict ingredient quantities from recipe contexts