Amodal Targets for Synthetic Scans: Teaching CAD-Trained 3D Encoders to Recognize Real, Partial Point Clouds Zero-Shot

Recent 3D recognition models learn open-vocabulary categories by matching point clouds of CAD models to the image and text embeddings of CLIP, a vision-language model trained on web images and captions. These models recognize thousands of categories on clean CAD benchmarks, but they fall apart on real scans, which are one-sided, noisy, and surrounded by clutter. We propose a way to keep the scale of synthetic CAD data while training on what a real sensor would actually see. During pre-training, each CAD shape is passed through a virtual depth scanner that produces a partial, noisy, cluttered point cloud. That partial scan is then aligned to the CLIP embedding of the whole object: the average over all rendered views, together with its caption. The encoder never sees a real scan in training. It learns that "a mug seen from one side, with a table edge in the crop" should map to the same point in language space as the complete mug.

The claim is narrow and testable. The persistent zero-shot sim-to-real drop of CAD-aligned 3D encoders is mostly a partiality and clutter gap, not a texture or asset-diversity gap. It can be closed without any target-domain data if the synthetic 3D input is made realistic and the semantic target stays amodal, meaning it describes the complete object rather than the visible fragment. The key ingredient is this asymmetry: a degraded 3D input paired with a complete semantic target. We isolate it against the cheaper option of random point-dropout augmentation.

Why this gap, and what is already known

Two lines of work set up the problem. ULIP, ULIP-2 and OpenShape align complete, uniformly sampled CAD point clouds with frozen CLIP encoders. They use rendered views and generated captions, and OpenShape scales this to 876k shapes. These papers supply our training recipe and motivate the problem. OpenShape reports 46.8% and ULIP-2 50.6% zero-shot top-1 on Objaverse-LVIS, yet OpenShape explicitly notes a sharp drop on the real ScanObjectNN scans. Its own finding that category diversity matters more than raw volume suggests that more CAD shapes alone will not fix a sensor gap. Revisiting Point Cloud Classification found that ModelNet40-trained classifiers fall below 50% on ScanObjectNN. That paper shows the gap predates vision-language pre-training.

The sim-to-real literature supplies the other half. ASM shows that synthetic LiDAR is "integral": every ray returns a point. It also shows that feature discriminators exploit the clean-versus-dropout difference as a shortcut, and that injecting learned dropout helps, moving SynLiDAR→SemKITTI mIoU from 15.2% to 22.8%. SIRA-PCR shows that realistic density variation matters for registration. CTS identifies object-size and density shifts in detection. All three are closed-vocabulary and single-modality. ASM and CTS also require unlabeled real target scans during training. We take their lesson that the sensor layer matters, but not their dependence on target data.

The closest connecting work is SynVL3D. It pre-trains a 3D-language model on ProcTHOR scenes, then removes the domain gap with a three-way adversarial discriminator during fine-tuning on real data. Our proposal differs in three ways: no real data at any stage, object-level open-vocabulary recognition rather than grounding, and a fix applied to the input and target of the contrastive loss rather than an adversarial loss on features. CLIP² takes the opposite route: it drops synthetic data and mines 1.1M real triplets by lifting 2D detections into LiDAR and RGB-D scans, reaching 61.3% zero-shot on SUN RGB-D. It is the strongest argument against our premise, and we treat it as a target to match. TeGA acts as a constraint. It shows that adding filtered Point-E shapes lifts ScanObjectNN zero-shot from 46.5% to 51.1%, but that fully synthetic generated data collapses. We read this as a warning that realism in the input geometry matters. That is exactly the variable we control. EUREKA's amodal retail work shows that synthetic scenes with ground-truth hidden geometry can teach occlusion reasoning that transfers to real images, reaching 37.10% AP on Real-IRAIS from synthetic-only training. We carry that "the simulator knows what is hidden" idea into open-vocabulary semantics.

Core Claim

Let $x$ be a CAD shape with caption $t$. Let $S(x;\phi)$ be a virtual scan of it under scanner settings $\phi$: viewpoint, sensor noise, dropout, and clutter. Let $f$ be the 3D encoder. The amodal target is

$$z(x) = \frac{1}{V}\sum_{v=1}^{V} g_{\text{img}}(r_v(x)),$$

where $r_v(x)$ is the $v$-th of $V$ rendered views of the complete shape and $g_{\text{img}}$ is the frozen CLIP image encoder. We train with the standard two-way contrastive loss between $f(S(x;\phi))$ and both $z(x)$ and the CLIP text embedding of $t$, drawing a fresh $\phi$ each epoch.

The "modal" alternative instead pairs the scan with the CLIP embedding of the single render taken from the scanner's own viewpoint. We predict that the amodal pairing transfers better to real scans. Real test labels name whole objects, and the modal target partially re-encodes viewpoint and visible extent as information the encoder must preserve. We also predict that physically structured partiality beats independent random dropout of points at a matched point budget. Real scans lose whole back-faces and grazing-angle regions, not random points.

Research Program

The main build is a virtual scanner that can be applied to any CAD repository. It ray-casts a mesh from a sampled viewpoint that mimics a handheld RGB-D sweep, which yields single-side or two-to-three-view partial coverage. It adds depth-dependent noise, drops returns at depth edges and grazing angles, and removes points on surfaces tagged as dark or transparent from mesh material metadata. This last step is a fixed rule inspired by ASM's dropout observation, not a learned module. The scanner then inserts clutter: a support plane and neighboring CAD objects inside a jittered crop box, imitating ScanObjectNN's background variants. Every corruption layer can be switched on or off independently. That makes the scanner an analysis tool as well as a data source.

The second build is the training variant itself. We use OpenShape's ensemble and Point-BERT backbone with OpenShape's text enrichment unchanged. The only differences are that virtual scans replace uniform samples on the 3D side, and that the semantic target is the amodal one above.

The analysis component decomposes the gap. We train the same encoder with each scanner layer enabled alone and in combination. Then we measure how much of the drop from clean CAD test data to real ScanObjectNN each layer recovers. This produces the first attribution, for vision-language 3D encoders, of which gap layer from the sim-to-real taxonomy (geometric simplification versus sensor noise) actually drives the failure.

A secondary stage tests whether the object-level gain carries into scenes. We swap our encoder in as the 3D-side classifier for class-agnostic Mask3D proposals, following the proposal/classification split in OpenMask3D. Points that lie outside any camera frustum then still receive features directly from geometry.

Baselines

The naive reference is OpenShape's released recipe retrained at our compute budget on clean, uniformly sampled CAD points. Everything else is measured as a change from it. For zero-shot ScanObjectNN accuracy of OpenShape we found no reported result on Lacuna, so we will measure it ourselves under the identical prompt set. The published anchor we can cite is TeGA's 51.1% zero-shot top-1 on ScanObjectNN, against a 46.5% baseline, from TeGA. Its backbone and training data are smaller, so it serves as a reference point rather than a like-for-like comparison.

The strongest cheap competitor is the same recipe with standard augmentation: random point dropout, jitter, and random crop, tuned with the same number of trials as our scanner parameters. This baseline isolates whether the gain comes from sensor-structured partiality or from generic robustness.

The modal-target ablation keeps our scanner but pairs each scan with its own-view render embedding. It isolates the amodal ingredient. A no-clutter ablation isolates the background layer.

The strongest real-data competitor is CLIP². It reports 61.3% zero-shot top-1 on SUN RGB-D, as given in CLIP². We evaluate on the same SUN RGB-D object protocol and backbone class, retraining CLIP²'s PointNet++ recipe only if our backbone differs.

Finally, PointTFA serves as a test-time comparison. It adapts frozen ULIP features with a labeled downstream support set without updating weights; its reported gain is +46.8% for PointNEXT over the frozen model, with no absolute zero-shot ScanObjectNN number in the file. We run it on top of both the naive encoder and ours with identical support sets. This checks whether our gain is complementary to support-set adaptation or redundant with it.

All pre-training runs share the data, backbone, number of steps, and hyperparameter search budget. Beating the field means clearly exceeding tuned dropout augmentation on both ScanObjectNN variants across three seeds, while staying within one point of the naive baseline on Objaverse-LVIS. Matching or exceeding CLIP² on SUN RGB-D without any real scans would be the headline result.

Why not just…?

Why not just add random dropout and jitter augmentation to OpenShape? This is the real threat, and the proposal is designed to be settled by it. Our bet rests on the observation from ASM that real sensor loss is spatially structured, which is why its structured masking beat generic alignment. If tuned random augmentation closes most of the gap on the PB_T50_RS variant, the method claim fails. In that case the gap-decomposition study remains the contribution, telling the field that the gap is cheap to fix. The matched-budget comparison decides this directly.

Why not just train on real scans, as CLIP² does? CLIP² needs synchronized cameras and inherits projection and clustering errors. It also cannot use the long tail of CAD categories: Objaverse-LVIS alone has 1,156. CAL shows the same real-data ceiling in a different task: pseudo-labels mined from real video reach 13.12 PQ† against 26.49 PQ† for supervised training, because unobserved sides cannot be recovered. If our synthetic-only encoder falls well short of 61.3% on SUN RGB-D, that is evidence for the real-data route. We would then test mixing CLIP²-style real triplets with virtual scans, rather than argue the result away.

Why not just adapt at test time with PointTFA or a kNN graph? Both need a labeled support set drawn from the target categories, which conflicts with open-vocabulary deployment. PGLLM shows that such graphs propagate errors when the base features are corrupted. We expect these methods to add on top of a better encoder rather than replace it. The PointTFA-on-both-encoders experiment measures whether that is true.

Why not just scale the CAD data further? OpenShape already scaled to 876k shapes and still reports the real-scan drop. Holding data fixed and changing only the scanner makes the source of any gain unambiguous.

Proposed Experiments

We evaluate zero-shot top-1 on ScanObjectNN OBJ_ONLY and PB_T50_RS with the 15 category names as prompts, and on SUN RGB-D objects and nuScenes objects following CLIP²'s protocol. ModelNet40 and Objaverse-LVIS serve as retention checks, since realistic inputs must not cost synthetic open-world accuracy. In the scene stage we report ScanNet200 open-vocabulary AP, head and tail, with Mask3D proposals. OpenMask3D reports 14.9 tail AP against 9.9 for the prior competitor, and we test whether 3D-only features recover objects that its frustum-limited features miss.

Three checks decide the proposal. The first is whether amodal pairing beats modal pairing on real scans; this is the core claim. The second is whether structured scanning beats tuned random dropout at a matched point budget; this is the method claim. The third is whether the layer-wise decomposition shows self-occlusion and clutter dominating over noise. Each result is informative on its own. If amodal targets win but random dropout matches the structured scanner, the paper becomes about the semantic target, not the simulator. If the decomposition shows noise dominating, it redirects effort toward learned sensor models like ASM's, now inside contrastive pre-training.