Models that turn engineering drawings into editable 3D CAD programs (CAD, or computer-aided design, is the parametric modeling software engineers use to specify manufacturable parts) are scored almost entirely on datasets built from public Onshape models. These datasets have been on the web for years. Web-trained vision-language models may therefore be recalling shapes they have already seen rather than reading the drawing. A second problem compounds the first. The metrics in use, such as command-token accuracy and whether the output builds at all, say little about whether the part would work. We propose a benchmark whose test parts do not exist anywhere until the evaluation round opens. Each part is generated procedurally from a secret random seed, and only its 2D drawings are published. The organizer commits to a cryptographic hash of the seed before release and reveals the seed afterwards. Anyone can then regenerate the ground-truth parts and re-run the functional grader, so no one has to trust the organizer's scores.
In two sentences: we replace "download the test set" with "receive fresh drawings of parts that were never on the internet, then have your outputs checked by a grader that anyone can re-run once the seed is revealed." This lets us measure, for the first time in drawing-to-CAD, how much of current performance is memorization. It also shows whether drawing-conditioned models fall off the same functional cliff that text-conditioned CAD models do.
Core Claim
Our central claim is that commit-reveal procedural generation is the right contamination defense for geometric engineering tasks. It works where password-protected archives and zero-knowledge proofs do not. With it, we expect to find a measurable memorization gap in web-pretrained models on existing drawing-to-CAD data, and a large gap between build success and functional success once drawings specify mating parts.
Two groups of prior work frame the claim. The first motivates it. Stop Uploading Test Data in Plain Text argues that encrypted archives and restrictive licenses only slow crawlers down, and that a single leak spoils a benchmark for good. DyVal shows that freshly generated reasoning instances expose memorization, with phi-1.5 and WizardMath-13B scoring near zero. However, DyVal only generates discrete arithmetic and logic problems.
The second group constrains the design. P⁴Bench keeps answers secret using zero-knowledge proofs, which let a solver prove it has a correct answer without revealing it. That only works for NP-style problems whose check fits a finite-field arithmetic circuit, such as SAT and discrete logarithm. It does not cover checks like "these two gears mesh without colliding under rigid-body simulation." We therefore deliberately do not pursue zero-knowledge proofs for continuous geometry. That would be the obvious move at this intersection, but it would force the grader into a circuit it cannot express. Commit-reveal keeps the grader arbitrary, including a Blender physics simulation, at the cost of making each round single-use. Procedural generation makes that cost acceptable.
Research Program
The generator. We build a public, versioned generator that turns a seed into three linked artifacts. The first is a parametric CAD program in the sketch-and-extrude vocabulary of DeepCAD, extended with holes, fillets and patterns. The second is the four vector drawings used by Drawing2CAD: three orthographic views plus one isometric, as SVG paths. The third is a functional specification.
The generator samples a part as a small graph of features with geometric constraints. This follows the constraint-graph view of SketchGraphs and the part-decomposition graphs of Graph-CAD. Depth and branching are controllable, the way DyVal controls graph depth. We fit the generator's static tier to DeepCAD's marginal statistics, such as the number of sketch loops, the number of extrusions and the aspect ratios, so that fresh parts are matched in difficulty to the public ones.
A functional tier generates pairs of parts that must mate. Examples are a shaft and bore with a specified clearance fit, a bracket whose holes must align with a plate, and a pair of spur gears at a given center distance. Dimensions are annotated on the drawings the way an engineer would annotate them. Grading reuses the check types of CAD-bench: dimensional tolerance checks, boolean volume difference against the regenerated reference, and rigid-body simulation of motion transmission.
The protocol. At the start of a round, the organizer publishes a commitment $c = H(s)$, where $H$ is a cryptographic hash and $s$ is the round seed. The seed itself is derived as $s = H(k \,\|\, b_t)$, combining an organizer secret $k$ with a public randomness-beacon value $b_t$ drawn after the commitment deadline, so the organizer cannot search for favorable seeds. The drawings for $G(s)$, where $G$ is the public generator, are then released.
Participants upload their output CAD files, or hashes of them, before the round closes. After the round, $k$ is revealed. Anyone can recompute $s$, regenerate the ground truth, and re-run the open-source grader on the committed outputs.
This removes the "trust the private API" problem that P⁴Bench raises about hosted benchmarks such as GSM1k (A Careful Examination of Large Language Model Performance on Grade School Arithmetic). The one residual trust point is an organizer secretly leaking $k$ before the reveal. Splitting $k$ across several organizers with threshold secret sharing reduces that risk, and we state it plainly as a limitation.
Input styles. Each part is rendered as clean SVG and as sparse, perturbed strokes. For the stroke version we use the non-photorealistic rendering idea from Fine-Tuned but Zero-Shot 3D Shape Sketch View Similarity and Retrieval. This tests the known weakness of patch-based vision models when strokes are sparse.
Baselines
The naive reference point is the status quo: score models on the public CAD-VGDrawing test split and treat that as their drawing-to-CAD ability. Every other comparison is measured as a gap from this number.
The strongest dedicated method is Drawing2CAD, a dual-decoder transformer that maps SVG commands to CAD commands. It reports 82.43% command accuracy, 76.09% parameter accuracy, and a 20.31% invalidity ratio with four views on CAD-VGDrawing (Drawing2CAD). We retrain it from scratch on the DeepCAD training split with the published recipe. Because it never saw web data, it serves as a clean control: any drop it shows on fresh parts measures distribution shift, not contamination.
The strongest general-purpose competitors are frontier multimodal models and agent harnesses that write CAD code from the drawings. On text-specified CAD-bench tasks, the best standalone model scored 59.9% overall and the best agent harness 67.7%. Standalone models averaged 58.5% on hard static tasks but only 3.7% on functional tasks (CAD-bench). We found no reported result on Lacuna for vision-language models on CAD-VGDrawing, so we establish that number ourselves. All models get the same four views, the same CadQuery/Blender tool access, the same number of retries, and the same token budget.
We adapt Graph-CAD, which reports 0.8943 geometric constraint satisfaction on CADBench Wild from text, using its released code (SCUT-MMPR/Graph-CAD). We replace its text encoder with a drawing encoder so it can take drawings as input. This tests whether its graph intermediate transfers to drawings.
A retrieval baseline returns the nearest DeepCAD training model to the input drawings, using Pivoting-style embeddings. It is a pure-memorization probe.
The key ablation isolates freshness. We compare three versions of the same round: fresh generated parts, public DeepCAD parts re-rendered with new styles and viewpoints, and fresh parts from the unmatched generator.
"Beating" has different meanings for the benchmark and for methods. The benchmark succeeds if it separates a memorization gap from a shift gap with confidence intervals that exclude zero. A method beats Drawing2CAD only if it wins on functional pass rate on fresh parts, not on token accuracy.
Why not just…?
Why not just keep a private held-out split of CAD-VGDrawing behind an evaluation server? The drawings would be private, but the CAD models they come from are public Onshape and DeepCAD files. A model that has memorized the geometry is still contaminated, however the drawings are hosted. A private server also requires permanent trust in the organizer, and P⁴Bench and the "Stop Uploading" audit, which found 4.2 uncontrolled citations per official submission, both argue that shadow use undermines hosted benchmarks. The re-rendered-public arm of our ablation tests this directly. If web-pretrained models score as well on re-rendered public parts as on fresh ones, this alternative is sufficient and our protocol adds only verifiability.
Why not just extend P⁴Bench's zero-knowledge proofs to geometry? Tolerance checks on a triangle mesh could perhaps be compiled into a proof circuit. Rigid-body simulation and B-rep boolean operations are far beyond any practical proof system. P⁴Bench itself also notes that a proof shows a solution was found, not that the model found it. Commit-reveal gives public re-gradability with an arbitrary grader. We lose only answer secrecy after the round, which procedural freshness makes irrelevant.
Why not just run post-hoc contamination detection with perplexity or n-gram overlap? Training corpora of closed models cannot be audited, and memorized geometry need not appear as verbatim tokens. The retrieval baseline and the difference-in-differences design measure memorization behaviorally, without any access to the training data.
Why not just prompt frontier vision-language models with CAD documentation? TheoremQA found that supplying definitions did not significantly help models apply them (TheoremQA), and CAD-bench shows that build success hides functional failure. We run documentation-augmented prompting as an explicit arm. If it closes the functional gap on fresh parts, that is a finding we will report, not argue around.
Proposed Experiments
Memorization gap. The first experiment is a difference-in-differences estimate. For each model family we compute the accuracy drop from CAD-VGDrawing test to matched fresh parts. We then subtract the from-scratch Drawing2CAD drop, which estimates pure distribution shift. The remainder is our memorization estimate. It is corroborated if the retrieval baseline's accuracy correlates with that remainder across public subsets. The decisive check is the sign and size of the residual. A near-zero residual would mean current drawing-to-CAD results are trustworthy, and would itself be a useful negative result.
Functional cliff under drawing input. The second experiment reports build rate, dimensional pass rate, volume intersection-over-union, and simulated-motion pass rate on the functional tier. It tests whether drawings, which carry explicit dimensions, soften the drop CAD-bench observed from 58.5% on hard static tasks to 3.7% on functional ones, or leave it unchanged. We also sweep constraint-graph depth to see whether failures grow with depth, as DyVal found for reasoning.
Protocol integrity. The third experiment audits the protocol itself. We run three public rounds and have outside volunteers independently regenerate every round from the revealed seed. Success means they reproduce every grade bit-for-bit. We also measure generator and grader cost per instance, to show that fresh rounds are cheap enough to run at least quarterly.
Stroke sparsity. The last experiment compares clean-SVG and sparse-stroke renderings of the same parts to quantify how much of the gap comes from reading the drawing rather than from CAD knowledge.
One threat cuts across all four experiments. The generator's code is public, so a lab could train on its distribution or search its grammar by analysis-by-synthesis. We treat that as a legitimate but separate "generator-aware" track, reported apart from the main leaderboard. Its size shows how much of the task a strong prior over part shapes can solve.
References
- Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- $\text{P}^4$Bench: Contamination-Proof, Publicly Verifiable, and Privacy-Preserving LLM Evaluation via Zero-Knowledge Proofs
- DeepCAD: A Deep Generative Network for Computer-Aided Design Models
- Drawing2CAD: Sequence-to-Sequence Learning for CAD Generation from Vector Drawings
- SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Design
- Learning Hierarchical and Geometry-Aware Graph Representations for Text-to-CAD
- CAD-bench: Benchmarking Language Models on Functional CAD Generation
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- Fine-Tuned but Zero-Shot 3D Shape Sketch View Similarity and Retrieval
- Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting
- TheoremQA: A Theorem-driven Question Answering Dataset