Starving the Condition: Measuring and Detecting Benchmark Contamination in 3D Latent Diffusion

Modern 3D shape generators are trained on hundreds of thousands of scraped or proprietary meshes and then scored on public shape collections that may well sit inside those training piles. Language-model research has spent years learning how badly leaked test items inflate scores. 3D generation has not yet asked the question. This note proposes a two-part study. The first part is a controlled experiment that measures how much a leaked test shape improves a model's reported score. The second is an audit that detects leakage using only the model's outputs, with no access to its weights or training data. The audit rests on one intuition: a model that has memorized a shape can rebuild it from almost no evidence, so its output quality should barely fall as the input is starved.

The setting is latent diffusion for 3D. A diffusion model is a generator that learns to turn random noise into samples by removing the noise step by step. In the latent version, this denoising runs on a compressed code produced by a separately trained autoencoder, rather than on raw voxels or meshes. That two-stage design is what makes 3D contamination different from the text case. A test shape can leak into the autoencoder, into the diffusion model, or into both, and each leak inflates a different benchmark number.

Core Claim

We claim two things. First, in conditional 3D latent diffusion, contamination causes a measurable and stage-specific inflation of standard metrics. A leak into the autoencoder should mainly inflate reconstruction scores such as IoU and Chamfer distance. A leak into the diffusion stage should inflate conditional generation from partial inputs. Second, contamination can be detected from outputs alone. For each test shape, we trace how output quality degrades as the conditioning input is thinned: fewer input points, a single occluded view, or a vaguer caption. Memorized shapes produce a flatter curve than complexity-matched unseen shapes. We call this the starvation curve.

The closest prior work motivates this detector. Detecting, Explaining, and Mitigating Memorization in Diffusion Models showed that in Stable Diffusion, memorized prompts make the text condition overwhelm the noise at the first denoising step. They detect this through the gap between conditioned and unconditioned noise predictions. Their method requires weights, operates on 2D images, and targets copyright rather than benchmark validity. The starvation curve is a black-box, output-space version of the same "condition overrides noise" signal, adapted to geometric inputs. We also test their white-box metric directly in 3D latent spaces as a baseline.

Research Program

The method comes from open two-stage 3D architectures. We use the set-of-latent-vectors autoencoder and transformer diffusion of 3DShape2VecSet. We add point-cloud and image conditioning in the style of ShapeR, which trains a rectified-flow transformer on the same kind of latents from sparse SLAM points, images and captions.

The training corpus is an Objaverse subset (Objaverse). We first scrub it of near-duplicates of ShapeNetCore test shapes (ShapeNet) by nearest-neighbor Chamfer search after canonical alignment. We then reinject a random half of a held-out ShapeNetCore test pool at duplication counts of 0, 1, 4 and 16. The injection goes into the autoencoder only, the diffusion model only, or both. The untouched half serves as the clean control.

This factorial design gives the causal quantity of interest. Inflation is the difference in a metric between injected and clean test shapes under the same model. Because the design crosses leak stage with duplication count, we can say which benchmark numbers are vulnerable to which kind of leak.

Two theory papers constrain what to expect. Losing dimensions predicts that high-variance directions memorize first. In 3D this suggests that coarse layout, such as overall silhouette and symmetry, will be recovered under starvation before fine surface detail. We therefore report coarse and fine geometric error separately rather than a single Chamfer number. Manifold Generalization Provably Proceeds Memorization in Diffusion Models and Local Coverage Governs Memorization in Diffusion Models imply that single-copy leaks in densely covered categories may leave no trace. A null result at duplication count 1 for chairs, for example, would be informative rather than a failure. For this reason each audit states its statistical power at every duplication level, following the spirit of When Is Benchmark Contamination Detectable?.

The main confound is that simple shapes also have flat starvation curves. We address it in two ways. First, every test shape is compared against unseen shapes matched on category, part count and a surface-complexity statistic. Second, we fit the detector on the difference between the target model's curve and that of a small reference model trained on clean public data.

Baselines

The naive reference point is to take reported benchmark scores at face value. Concretely, the clean control half of the test pool, scored by the same model, is what an uncontaminated evaluation would produce. Any inflation we measure is relative to it.

The strongest white-box competitor is the first-step noise-discrepancy metric of Detecting, Explaining, and Mitigating Memorization in Diffusion Models, ported to 3D. We compute it in the VecSet latent space, comparing the model's noise prediction with the point-cloud or text condition against its prediction with an empty condition. It uses the same trained models, the same test shapes, and the same threshold-calibration budget on clean shapes that our detector gets. In 2D it reports an AUC of 0.960 and a true-positive rate of 0.760 at 1% false positives on Stable Diffusion memorization prompts. We found no reported result on Lacuna for any memorization or contamination detector on 3D latent diffusion.

The second white-box baseline is a denoising-loss membership test, which flags a shape if the model's training loss on it is unusually low. The oracle baseline is nearest-neighbor retrieval against the known training set. It is the ground truth when the corpus is public, and it is the upper bound our black-box audit must approach.

Our starvation-curve detector counts as beating the white-box metrics if it matches their AUC within the confidence interval while using outputs only. It clearly wins if it exceeds them at low duplication counts. The key ablation replaces the curve with output quality at full conditioning only. This isolates whether the gain comes from starving the input.

For the inflation measurements, the relevant published anchors are the autoencoder IoU of 0.965 and generation surface-FPD of 0.76 on ShapeNet-v2 reported by 3DShape2VecSet. Our clean-corpus reproduction should land near these before any contamination is added.

Why not just…?

Why not just deduplicate the training set against the test set? When the corpus is public, this works, and we use it as our oracle. It is not an option for outsiders auditing models trained on private collections, such as the 600K-mesh corpus behind ShapeR. It also misses leaks where the same object appears with different triangulation, scale or pose, much as n-gram filters miss rephrased text (Rethinking Benchmark and Contamination for Language Models with Rephrased Samples). We test the second point directly by injecting remeshed and re-posed copies alongside exact copies.

Why not just render views and run 2D copy detection or CLIP similarity? GT23D-Bench shows that 2D proxies track 3D geometric quality poorly. A 2D image-matching score would also flag any generic chair that resembles a test chair. The real threat is a render-based version of our own curve. We include it as an ablation by scoring rendered silhouettes instead of 3D geometry, and we report whether it loses power.

Why not just use the white-box noise-discrepancy metric? It needs weights, which most deployed 3D generators do not release. It was also designed for text prompts, and it is unclear whether a point-cloud condition dominates the first step in the same way across an unordered latent set. If the metric transfers cleanly, that is itself a finding, and it becomes the recommended tool whenever weights are available. Our contribution then narrows to the black-box setting and to the stage-resolved inflation measurements.

Why not just evaluate on objects captured after the training cutoff? Newly captured scans differ in sensor noise and completeness. ScanNet++ reports PSNR falling from 25.02 to 17.70 when training moves from DSLR to iPhone captures. Any score gap would therefore mix contamination with domain shift. We do use ShapeR's 178-object in-the-wild set as a probably clean real-world reference for the audit's false-positive rate.

Proposed Experiments

The first experiment is the factorial injection study described above. For each leak stage and duplication count, it reports the inflation in IoU, coarse and fine Chamfer distance, and surface-FPD, with bootstrap intervals over test shapes.

The second is detection. We compute AUC and true-positive rate at 1% false positives for the starvation curve, the ported noise-discrepancy metric, loss-based membership, and full-conditioning quality alone. Results are broken down by duplication count, by leak stage, and by exact versus remeshed copies.

The third applies the calibrated black-box audit to publicly released Objaverse-trained generators, scored on ShapeNetCore and GT23D-Bench evaluation items. For each model it reports an estimated contaminated fraction with a stated power level, rather than a yes-or-no verdict.

Three decisive checks determine whether the project succeeds. First, autoencoder-only leaks must inflate reconstruction but not sparse-input generation, which confirms that the stages matter. Second, the starvation curve must separate injected from matched clean shapes at duplication count 4 or lower. Third, that separation must survive the complexity-matched control. If the curve fails the complexity control, the method is measuring shape simplicity, not memorization, and the claim is wrong.