Text-to-CAD systems read a written description of a part, such as "a 40 mm plate with four corner holes," and produce the sequence of modeling commands a CAD program would run to build it: sketch, extrude, cut. They now report strong scores on public benchmarks. The test sets behind those scores, DeepCAD and its Text2CAD annotations, have been freely downloadable for years. So a low geometric error could mean the model reasons about shape, or it could mean the model has seen the exact part before and is recalling its command string.
We propose a simple test for telling these apart. Take a benchmark part and change one fact in both its program and its description, for example 40 mm becomes 55 mm, or the extrusion flips to the other side. The CAD kernel then gives the exact correct answer for the edited part, so no human or image editor is needed. The test asks whether the model follows the edit or snaps back to the original part. A model that reasons follows the edit. A model that memorized the item is pulled toward the answer it saw in training, and it does so even though the edited description differs from the original by a single token.
Core Claim
Accuracy drops alone cannot separate memorization from ordinary prompt brittleness. The direction of the error can. Under a meaning-changing edit, a brittle but clean model makes errors that are scattered. A contaminated model makes errors that point toward the original target.
CAD is an unusually good place to measure this, for two reasons. First, the output is an executable program, so we can read off exactly which parameter value the model used. Second, the kernel guarantees that the counterfactual target is correct.
We therefore pair two kinds of perturbation per benchmark item:
- Meaning-preserving paraphrases, which measure brittleness.
- Meaning-changing single-fact edits, which measure memorization.
We report two direction-sensitive quantities alongside the usual invalid rate and Chamfer distance (the average distance between points on the generated surface and the reference surface):
- Stale-value rate. For numeric edits, this is the fraction of outputs that use the old value instead of the new one.
- Attraction score. For geometric edits, let $\hat{y}'$ be the model's output on the edited prompt, $y$ the original target, and $y'$ the edited target. The score is
$$a = \frac{\mathrm{CD}(\hat{y}', y')}{\mathrm{CD}(\hat{y}', y) + \mathrm{CD}(\hat{y}', y')},$$
where CD is Chamfer distance. A value of $a$ near 1 means the output lies closer to the original part than to the part that was asked for. The central prediction is a clean dissociation between the two groups of models:
- Contaminated models show high stale-value rates and high attraction on items they were trained on.
- Clean but brittle models degrade under paraphrase but keep $a$ near or below 0.5.
Relation to Prior Work
The detection principle comes from Contamination Detection for VLMs using Multi-Modal Semantic Perturbation. That work builds counterfactual versions of multiple-choice visual questions that keep the reasoning the same but change the answer, and it edits images with Flux under ControlNet. Deliberately contaminated LLaVA and Qwen2-VL models lost 8–32% accuracy, while clean models gained about 31% on MMStar.
That gain is exactly the confound we want to remove. The diffusion edit made the images easier, so the perturbed set was not matched in difficulty to the original. The authors also note that Flux struggles with fine geometry and text. Our proposal differs from theirs in three ways:
- We replace generative image editing with exact edits to the program, checked by the kernel.
- We work on open-ended generation rather than multiple choice.
- We read the direction of the error rather than only its size.
The brittleness side comes from Brittlebench and PromptBench. Brittlebench attributes up to half of performance variance to meaning-preserving perturbations. PromptBench offers character-, word-, sentence- and semantic-level attacks. Both are text-only and multiple-choice. Brittlebench's finding matters here as a warning: a paraphrase-induced drop is expected even in clean models, so it cannot by itself indicate contamination. We reuse their perturbation types for our paraphrase arm and adapt their variance split.
DyVal generates fresh reasoning problems from graphs. Models strong on static benchmarks, such as phi-1.5 and WizardMath-13B, scored near zero on these fresh problems. A CAD program is naturally a dependency graph (sketch, then extrude, then fillet, then cut), so we borrow this idea to generate never-published parts as a clean reference.
The models under test come from two papers:
- Text2CAD supplies the benchmark, with about 170k DeepCAD parts and 660k prompts at four abstraction levels. It reports a median Chamfer distance of 0.37 against DeepCAD's 32.82 and an invalid rate of 0.93%.
- Plan then Act uses a Qwen3-8B planner and a numeric "actioner" decoder that looks up dimensions from the text. It cuts GPT-4o's 70.35% invalid rate on abstract L0 prompts to 0.63%.
Plan-then-Act's two-level design lets us ask where snapping back originates. It could come from the planner recalling a remembered plan. It could also come from the actioner copying a remembered number despite a new number in the text.
Research Program
Building the counterfactual suite. For each Text2CAD test item at levels L2 and L3, where descriptions carry concrete geometry, we apply one edit to the ground-truth program. The edit types are:
- scaling a single dimension outside the original value's rounding neighborhood;
- flipping an extrusion direction;
- changing hole count;
- swapping cut and join.
We execute the edited program to confirm it produces a valid solid. We then rewrite only the corresponding words or numbers in the description. The edited prompt therefore keeps near-maximal overlap with the original, which is the condition under which a memorizer is most tempted to snap back. L0 prompts, such as "a water drop," carry no editable facts, so they receive only the paraphrase arm.
Difficulty control. We match each edit to the original item by command count and operation types. We also include a DyVal-style set of freshly generated programs of matched depth. This set is guaranteed unseen, so it provides the clean-model reference distribution for $a$ and for the stale-value rate.
Controlled contamination. To get ground-truth labels, we fine-tune Qwen3-8B, in both a direct-generation version and a Plan-then-Act-style planner version, on the Text2CAD training split. For each copy we add a known fraction of test items, repeated 1, 3 or 10 times. This follows the protocol of Park et al.
Controlled brittleness. Separately, we train a clean model on a single phrasing template per part. This model should be brittle to paraphrase but uncontaminated. It is the key control that a drop-only detector cannot pass.
Auditing released systems. Finally, we audit released and API systems: GPT-4o, the released Text2CAD model, and Plan-then-Act if its code (listed as QiferG/Plan-then-Act, currently "coming soon") becomes available. For each, we report per-level stale-value and attraction rates together with the fraction of the published score that survives counterfactual editing.
Secondary extension to graphic layouts. A short extension checks whether the principle carries beyond CAD. On the CreatiDesign 1,000-sample benchmark from CreatiDesign, we move one bounding box and measure the overlap (intersection over union) of the rendered subject with the old box versus the new one.
Baselines
Static score (do nothing). The naive reference is the unmodified benchmark score. The Text2CAD paper reports a median Chamfer distance of 0.37 and an invalid rate of 0.93%, and Plan-then-Act reports a 0.63% invalid rate at L0. We re-run each model on the original items with the same decoding settings. We report the gap between this score and the counterfactual-suite score, rather than treating the static score as a detector.
Text overlap and n-gram matching. This checks the test prompts and command strings against the training data that is available. We apply it wherever training corpora are open, which is our fine-tuned models and Text2CAD. It is the cheapest competitor. It counts as beaten if our item-level detector has higher area under the ROC curve (AUROC) in the controlled-contamination setting. For API models it simply cannot be run, and we report that directly.
Token-likelihood detection. Following the observation in Studying and improving reasoning in humans and machines that contaminated models predict test tokens with near-certainty, this baseline scores each item by the likelihood the model assigns to the ground-truth command sequence. It is run on open-weight models only, with the same items.
Accuracy-drop detection (strongest competitor). The strongest existing method is Park et al.'s signal: flag an item or model when performance falls under counterfactual perturbation. We implement it on exactly our counterfactual items, using the drop in validity and Chamfer distance, with an identical threshold-tuning budget on a held-out calibration split.
Its reported figures are for multiple-choice visual questions: drops of 8–32% for contaminated models and a ~31% gain for clean models on MMStar. No reported result was found on Lacuna for contamination detection on any CAD or design generation benchmark. We beat this baseline if, at matched false-positive rate, our directional scores separate contaminated from clean-but-brittle models, a comparison the drop signal should fail.
Ablation. The ablation that isolates the key ingredient keeps the full suite but removes the direction term. It scores only the increase in $\mathrm{CD}(\hat{y}', y')$ and never compares against $y$. If this ablation matches the full detector, the directional idea contributes nothing.
Why not just…?
Why not just deduplicate against training data with n-grams? Park et al. report that text-only detectors are inconsistent for multimodal models because they ignore what was memorized outside the text channel. In CAD, the memorized object is a shape that can be reached from many phrasings, including LLM-generated rewrites like Text2CAD's own annotations. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples is the standard warning that rephrased leaks evade string matching. More decisively, overlap search needs the training corpus, which is unavailable for GPT-4o and most base models. The controlled experiment quantifies the gap where both methods can be run.
Why not just build a fresh procedurally generated CAD benchmark, DyVal-style? This is a real threat, and we take it seriously. If the only goal were a clean capability number, fresh generation would suffice, and we build such a set ourselves. It cannot, however, say which published items leaked or how much of an existing reported score is recall. It also changes the part distribution: procedurally generated parts do not look like DeepCAD's human-designed parts, so a score gap mixes contamination with distribution shift. Counterfactual edits stay on the original items. We settle the question empirically. If the gap between fresh and static scores explains model rankings as well as our stale-value rate does, and matches it item for item, the audit adds little and we will say so.
Why not just use likelihood or perplexity? It needs log-probabilities, so it excludes API models. On The Fragility of Benchmark Contamination Detection in Reasoning Models reports that such heuristics break for reasoning models, and Plan-then-Act-style planners fall in that class. It will be compared head-to-head on open weights.
Why not just use Park et al.'s accuracy drop? Brittlebench shows that meaning-preserving perturbations alone move scores by up to 12% and flip rankings in 63% of cases. A drop is therefore expected in clean but brittle models. The template-trained brittle control tests exactly this: a drop-based detector should flag it, and a directional one should not.
Decisive Checks
Three results would decide whether the proposal holds up.
Calibration. In controlled contamination, we expect the stale-value rate and attraction score to rise with leak repetition and to reach higher item-level AUROC than both likelihood and accuracy drop.
Brittle control. This is the make-or-break test. The template-trained model should show paraphrase degradation comparable to contaminated models, yet its attraction score should stay at the fresh-set reference level. If attraction rises for this model as well, the directional signal is merely measuring brittleness, and the core claim fails.
Localization. In the two-level system, we swap planner outputs between the original and edited prompts. If stale values persist when the planner's plan is correct, snapping back lives in the numeric actioner. That would bear directly on the claim that the actioner's dimension lookup "reads" the text. The layout extension is reported only as a transfer check. A null result there would narrow the claim to parametric generation rather than overturn it.
References
- Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
- Brittlebench: Quantifying LLM robustness via prompt sensitivity
- PromptBench: A Unified Library for Evaluation of Large Language Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts
- Plan then Act: Bi-level CAD Command Sequence Generation
- CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design
- Studying and improving reasoning in humans and machines
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- On The Fragility of Benchmark Contamination Detection in Reasoning Models