Routing Synthetic Data to Geometry: Factorized Sim-to-Real Calorie Estimation

A plate's calories are, roughly, how much food there is multiplied by how energy-dense that food is. Computer graphics can cheaply produce realistic examples of the first factor and only poorly produce the second. A rendered heap of rice in a bowl has believable shape and depth. It does not have believable sheen, translucency, or the look of food cooked in oil, and appearance is exactly where hidden fats and sugars show up. We propose a calorie estimator split along this line. A volume branch reads only the depth image and is trained mostly on synthetic scenes, with learned sensor noise added so that clean renders look like real depth-camera output. A separate density branch reads color and is trained only on real photographs. Its internal representation is built to capture what volume cannot explain, which is where invisible ingredients should end up.

This matters because the best published real-world system, Nutrition5k, shows that adding a depth-derived volume estimate cuts calorie error from 26.1% to 16.5%. The same paper notes that collecting weighed, depth-scanned dishes is very expensive. Synthetic data is the obvious way to scale, but prior food synthesis has either stayed in the classification setting or never been tested on a real quantity task. Our bet is that the synthetic-to-real gap in food is lopsided. Geometry transfers, appearance does not, so synthetic supervision should be sent only where it transfers.

Core Claim

We claim that factorizing log energy into a geometry term and a residual density term lets synthetic depth data improve real-world calorie and mass estimation under distribution shift, where unfactorized synthetic pretraining fails or helps less. We write

$$\log \hat{E} = \log \hat{V}(d, m) + \log \hat{\rho}(x, z),$$

where $\hat{E}$ is predicted dish energy in kilocalories, $\hat{V}$ is predicted food volume computed from the depth map $d$ and a food mask $m$, $\hat{\rho}$ is predicted energy per unit volume computed from the color image $x$, and $z$ is a low-dimensional latent code produced by the density encoder. The same structure with mass in place of energy gives a mass head. The novelty is not synthetic food data as such. It is the claim that the place to inject synthetic supervision is decided by where the rendering gap is small, and the experiments test that claim directly against the unfactorized alternative.

The second, smaller claim borrows from latent phenotyping. In REGLE, known clinical measurements are fed into a variational autoencoder's decoder so that the encoder is pushed to capture the residual variation those measurements miss. The authors show this residual code carries new biological signal. Our density code $z$ plays the same role: because volume is supplied separately, $z$ should encode composition, including ingredients the camera cannot see. We test this against the per-ingredient mass logs in Nutrition5k rather than asserting it.

Relation to Prior Work

Three food papers motivate the proposal and also limit what it can assume. Shape-Preserving Generation of Food Images builds a conditional GAN that swaps food classes inside a fixed shape and container (FID 22.82 on Food-101, mask IoU 0.91 on UEC-FoodPIX). It never trains a volume or mass model on its outputs, and its failures, such as liquids spread across flat plates and plate patterns bleeding into food, are appearance failures. We take this as evidence that synthetic appearance is the unreliable part and do not feed generated color images to any quantity head. Food Volume Estimation via View Synthesis trains on rendered depth of rigid objects and reaches 97.6% volume accuracy in the lab. It degrades when objects move off-center and does not handle mixed meals. We reuse its premise that depth geometry can be learned from renders, but we train on deformable, multi-component procedural scenes and evaluate on real cafeteria depth. DiSC-DS finds that diffusion-generated food images help long-tailed classification only when mixed with real photos through Mixup. This is a constraint: the synthetic appearance gap is large enough to need patching even for coarse labels, which supports keeping synthetic color out of fine-grained regression.

Two sim-to-real papers supply the method. PixelDA translates synthetic images toward real style while preserving geometric labels, and beats feature-level alignment on synthetic-to-real LineMod pose (23.5° versus 56.58° for DANN). ASM learns where to delete points from clean synthetic LiDAR scans so they fool a domain discriminator. It shows that plain feature alignment fails because the discriminator learns to detect noise rather than compare shapes, and it raises SynLiDAR→SemKITTI from 15.2% to 22.8% mIoU. We adapt ASM's learned, differentiable dropout to depth images. This targets the specific artifacts of the consumer depth camera used in Nutrition5k: holes on dark or specular food, edge noise, and missing values at plate rims. PixelDA-style translation is kept as a comparison.

From latent phenotyping, REGLE supplies the residual-conditioning design. iGWAS supplies a second ingredient. It trains a contrastive model that treats a person's two eyes as natural replicates sharing one genome. We treat different camera views and video frames of the same Nutrition5k dish as replicates sharing one composition, which makes $z$ insensitive to viewpoint and lighting. Cross-modal food embeddings such as Recipe1M and PITA learn composition, and in PITA's case relative ingredient proportions, but they cannot recover absolute scale. Our volume branch provides the scale they lack.

We do not run genetic association studies. No dataset in our material pairs food imagery with genotypes, so we use the phenotyping literature for its design principle and its validation logic, not its application.

Research Program

The first component is a procedural synthetic dataset of food scenes. We generate deformable heaps, piles, and layered components in bowls and plates of varying shapes, using soft-body or heightfield sculpting. Exact volume and per-pixel depth come for free. Rendered views match the overhead geometry and intrinsic camera parameters of the Nutrition5k rig. Color is rendered but used only for building the food mask during training, never as input to a quantity head.

The second component is a learned depth-corruption module in the style of ASM. It applies a Gumbel-Softmax mask, a differentiable way of choosing which pixels to drop, together with edge perturbation. It is trained adversarially so that corrupted synthetic depth cannot be told apart from real Nutrition5k depth by a discriminator that sees only depth. The discriminator never sees volume labels, so this alignment uses no real volume ground truth.

The third component is the volume branch, a depth-only encoder-decoder that predicts a completed height field and integrates it into $\hat{V}$. It is trained on corrupted synthetic depth with exact volume labels. On real dishes it receives only an indirect signal: the gradient through the product with density when the end-to-end energy loss is applied. That signal is scaled down by a weight that we sweep, so the branch stays anchored to geometry.

The fourth component is the density branch, a color encoder producing $z$ and a head producing $\hat{\rho}$. It is trained on real Nutrition5k dishes using energy and mass labels together with the detached volume estimate, plus an iGWAS-style contrastive loss across views of the same dish. Because volume is given, the energy loss can only be lowered by $z$ learning energy per unit volume.

Baselines

The naive reference point is the Nutrition5k RGB-only regressor, which reports 26.1% calorie error. We also report a mean-per-dish-type predictor so that every gain is read against the simplest possible estimate.

The strongest existing method is the Nutrition5k RGB-plus-volume-scalar model trained only on real data. It reports 16.5% calorie error and 26.2% aggregate macronutrient mean absolute error in Nutrition5k. It is the strongest competitor because it already uses depth and already factorizes volume implicitly, with the volume scalar computed from real depth. We re-implement it on our splits with the same backbone family and resolution, and give it the same tuning budget in trials and GPU-hours as our model. We first confirm that our re-implementation lands near the published number on the original split. We are not counted as beating it in-distribution unless we beat our own re-run by more than the spread across seeds. Our main claim, however, concerns the shifted splits described below, where we need a clear margin over the re-run.

The unfactorized synthetic baseline pretrains a single RGB-D network on synthetic renders, translated with PixelDA for color and passed through the same ASM corruption for depth, and then fine-tunes everything on real data. It isolates our central choice, which is sending synthetic data only to geometry, from the simple fact of having more data.

Three ablations isolate our key ingredients. The first replaces learned ASM corruption with hand-tuned random dropout and Gaussian depth noise. The second removes the synthetic data and trains the volume branch only through the real energy loss, which tests factorization without synthesis. The third removes the contrastive replicate loss from the density branch. For the hidden-ingredient analysis, the baseline is a linear probe on the penultimate features of the strongest re-run model.

Why not just…?

Why not just train the published RGB-D model on more real Nutrition5k data and skip synthesis? In-distribution, this may match us, and we say so plainly. Factorization is only worth its complexity if it helps where real data is thin. Our shifted splits hold out the largest portions, unseen containers, and dishes whose main ingredient never appears as a main ingredient in training. These directly test whether synthetic geometry covering wide volume ranges and shapes buys extrapolation. If the re-run real-only model matches us on those splits, the proposal's main claim is false.

Why not just pretrain on synthetic RGB-D and fine-tune end to end? This is the most serious threat, because fine-tuning might discard the synthetic appearance bias on its own. The DiSC-DS finding that synthetic food appearance needs Mixup even for classification suggests that appearance features learned from renders actively mislead. Still, fine-tuning could undo that. The unfactorized synthetic baseline settles the question under an identical data and compute budget. If it matches the factorized model on the shifted splits, we will report that the gain comes from synthetic data in general and not from routing.

Why not just add random depth noise instead of learning it adversarially? ASM's evidence is that learned, location-dependent dropout beats generic perturbation on LiDAR. The artifacts of food depth also depend on location, appearing on dark sauces, glossy surfaces, and rims. Even so, simple noise may be enough for a mostly overhead, near-field camera. The first ablation measures exactly this, and if hand-tuned noise matches learned noise we will adopt the simpler version.

Why not just use an off-the-shelf monocular depth estimator so no depth sensor is needed? The material notes that monocular depth has scale ambiguity that destabilizes absolute mass estimates. We include it as a deployment-shift condition: monocular depth rescaled by plate diameter is fed to every model's depth input. This tells us whether the volume branch's robustness to noise carries over to a phone-like setting.

Proposed Experiments

All experiments use Nutrition5k overhead RGB-D images and per-ingredient mass logs. Metrics are calorie, mass, and macronutrient mean absolute error as a percentage of the mean, with three seeds. There are four evaluation splits. The first is the original split, as a sanity check against the published 16.5%. The second is portion extrapolation, training on dishes below the 75th mass percentile and testing above it. The third is container holdout. The fourth is main-ingredient holdout. Each split is also run with monocular depth substituted for sensor depth. The decisive check is the portion-extrapolation and main-ingredient splits. The factorized model should reduce calorie error relative to both the re-run real-only model and the unfactorized synthetic model by more than the spread across seeds. Equivalence with either one counts as a negative result for routing.

A second check validates the volume branch on its own terms. Mass divided by predicted volume should be roughly constant across Nutrition5k dishes that consist of a single ingredient. We measure the spread of this ratio within each ingredient and expect it to be smaller for the volume branch with learned corruption than for the versions trained without it.

The third check tests the residual-latent claim. Using the ingredient logs, we compute each dish's mass of components that are typically invisible: oils, dressings, sauces, and added sugar. We then fit linear probes from $z$ and from the strongest baseline's features to that hidden mass, and to the fraction of calories it contributes. If the REGLE-style residual design works as intended, $z$ should predict hidden-ingredient energy better, especially among dishes of similar volume, where geometry offers no help. If it does not, the density branch is merely a calibration factor, and we will report that the hidden-ingredient problem PITA and Nutrition5k describe remains unsolved by this route.