When a 3D network keeps only a few hundred points out of a real depth scan, it has to decide which points matter. Learned samplers make that choice by favoring "fine structure": thin legs, sharp edges, small protrusions. On clean computer-aided design (CAD) models this works almost as well as keeping every point. On real scans it should fail in a predictable way. The isolated points a sensor fabricates, such as floating speckle returns from an active stereo camera or ghost returns from fog in front of a LiDAR, look locally like fine structure. A sampler that rewards fine structure therefore spends its scarce budget on sensor artifacts. We propose to fix this with supervision that physics-based sensor simulators already produce and nobody uses. A simulator knows, for every point it emits, whether that point lies on the true surface or was invented by the measurement process. We use that per-point record as a training target for a reliability score, and we fold the score into a learned sampler.
The core claim fits in two sentences. Physics-based sensor simulators produce, for free, a per-point label of "real surface versus measurement artifact." Training a downsampler to predict that label from surrounding context should keep low-point-budget models accurate on real scans, where today's samplers trained only for the task waste their budget on noise. The claim is narrow and falsifiable. If a sampler trained on the same simulated data with only the classification loss learns to avoid artifacts just as well, the extra supervision is unnecessary and the idea fails.
Where the gap is
Three lines of work meet here, and none of them covers the junction. Physics-based simulation of sensor noise improves sim-to-real transfer, but only as input data. Mesh2Point / QS³ simulates a speckle projector and stereo matching inside Blender. That alone lifts a DGCNN classifier from 46.7% to 64.0% on real DepthScanNet10, and pseudo-label self-training brings it to 72.4%. FOGGIFY simulates fog attenuation and backscatter on real LiDAR scans. Neither paper ever looks inside the network. Both discard the simulator's knowledge of which points are artifacts after rendering. These papers supply our data generators and motivate the proposal.
Learned downsampling supplies the method we modify. Learnable Skeleton-Aware 3D Point Cloud Sampling ranks points by local feature size, meaning distance to the shape's medial skeleton, and selects them with a differentiable Gumbel-Softmax step. It reaches 89.1% on ModelNet40 with 128 points against 89.5% with the full cloud. Its authors note that skeleton fitting is sensitive to noise and outliers. That observation is the failure we target. LIP makes the same "weight by local importance" move for 2D pooling, and it too was tested only on clean data.
Two negative results constrain the design. Li et al.'s corruption benchmark finds that k-nearest-neighbor outlier removal gives limited benefit and sometimes hurts, because attenuated real surfaces are sparse and get pruned along with the noise. Artifact detection therefore cannot be a context-free density filter. It has to use surrounding geometry, which is the local-ambiguity problem: a small isolated cluster could be a distant object or fog. ScanObjectNN's background-aware network shows that an auxiliary per-point foreground mask helps real-scan classification, raising PointNet++ from 77.9% to 80.2% on PB_T50_RS. That supports per-point auxiliary supervision in general. Its labels, however, are hand annotations of clutter rather than simulator-derived artifact labels, and it does not touch sampling. AdaptPoint learns sample-specific corruptions adversarially. Because those corruptions are not physically generated, it has no ground-truth notion of which points are fabricated.
Core mechanism
For each simulated point $i$, the simulator gives a residual $d_i$, the distance from the point to the ground-truth mesh. For FOGGIFY, it also gives a provenance flag marking soft-target backscatter versus hard-target return. From these we define a soft reliability target $y_i = \exp(-d_i^2/\sigma^2)$, set to zero for backscatter. Here $\sigma$ is on the order of the sensor's nominal depth precision.
A small head on the sampler's per-point features predicts reliability $r_i \in (0,1)$. The head sees features aggregated over a ball-query neighborhood. Ball query is chosen over k-nearest neighbors because Sun et al. found it resists background noise better. The sampler's selection logit becomes the skeleton-aware importance $a_i$ plus $\log r_i$. Points that are structurally interesting but probably fabricated are therefore suppressed multiplicatively. Training minimizes
$$ \mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_i \mathrm{BCE}(r_i, y_i), $$
where $\mathcal{L}_{\text{task}}$ is the classification loss after sampling and $\lambda$ weights the reliability term. At test time no simulator is needed. The reliability head runs on raw real scans, and the sampler keeps $k$ points.
Research Program
The work has three stages.
The first stage builds a provenance-tagged simulation set. We re-render ModelNet40 meshes through a Mesh2Point-style active stereo pipeline and log each point's residual to the mesh. In parallel, we run FOGGIFY on KITTI scans and log backscatter flags.
The second stage measures the problem before solving it. We train the skeleton-aware sampler and LIP-style importance scoring on this data with the task loss only. On held-out simulated scans we then measure the artifact budget fraction: the share of the $k$ selected points with $y_i < 0.5$. We compare it with the artifact rate in the full cloud. The hypothesis requires that this fraction exceed the base rate, meaning the samplers actively prefer noise. If it does not, the motivating failure is absent and we stop.
The third stage trains the reliability-weighted sampler and evaluates it on real data at budgets of 128, 256 and 512 points, using PointNet++ and DGCNN as classifiers. On real scans there are no artifact labels. We use ScanObjectNN's background masks as a proxy and report the fraction of sampled points that fall on clutter.
Baselines
The naive reference is farthest point sampling, which spreads points evenly by geometry, applied to the same SpeckleNet-style training data and the same classifier. It tells us whether learned sampling helps at all under real noise.
The most important baseline is the strongest competitor on the actual question: the skeleton-aware sampler of the CVPR 2023 paper, trained on exactly our provenance-tagged simulated data with the task loss only. It sees the same noisy inputs, the same compute and the same number of epochs. We will tune it over a grid of learning rate and temperature equal in size to ours. Checkpoints for all methods will be selected on a held-out simulated validation split, never on the real test set, following Goyal et al.'s warning about test-set feedback. This baseline is also the ablation that isolates our key ingredient, since it differs from our method only by the reliability supervision.
A second ablation keeps the reliability head but removes neighborhood context. The head sees only per-point features. This tests whether context is what separates artifacts from sparse real surfaces.
For full-resolution reference points we cite reported numbers. QS³ reaches 72.4% on DepthScanNet10 and SpeckleNet-only DGCNN reaches 64.0% (Chen et al.). On ScanObjectNN PB_T50_RS, BGA-PN++ reaches 80.2% and SimpleView 79.5%. AdaptPoint reports 78.3% mean corruption error on ScanObjectNN-C. Beating our method means two things. At 128 and 256 points it must exceed the task-only skeleton sampler by more than seed variance on DepthScanNet10 and ScanObjectNN, measured over five seeds. It must also close a measurable share of the gap to the matching full-resolution model. Recovering the SpeckleNet-only 64.0% with 256 points would be a meaningful outcome. No reported result was found on Lacuna for learned samplers at low point budgets on DepthScanNet10 or ScanObjectNN, so those numbers will be produced here.
Why not just…?
Why not just train the existing learned sampler on physics-simulated data and let the task loss teach it to avoid noise? This is the real threat, and it may well work partially. The classification loss only says that a wrong answer happened, not which kept points caused it. At 128 points a few wasted points may not change the label on simulated data, while still mattering on harder real scans. The task-only sampler baseline settles this directly. If its artifact budget fraction and real-scan accuracy match ours, we will report that the supervision is redundant.
Why not just remove outliers with a k-nearest-neighbor filter before sampling? Li et al. found this gives limited benefit and can degrade accuracy, because attenuated or distant real surfaces have the same low density as noise. We still include filter-then-FPS and filter-then-skeleton-sampling as baselines, with the filter threshold tuned on simulated validation data.
Why not skip sampling and use stronger pretraining such as ULIP? ULIP reaches 89.4% on ScanObjectNN with PointMLP at full resolution, which is far stronger than anything we target. Our question is about the low-point-budget regime, which pretraining does not address. The two approaches are complementary. As a secondary experiment, we will place our sampler in front of a ULIP-pretrained encoder to test whether the gains persist on top of a strong representation.
Proposed Experiments
The primary transfer is ModelNet40 rendered through simulated active stereo, tested on real DepthScanNet10 and on ScanObjectNN OBJ_ONLY and PB_T50_RS. We also evaluate on ScanObjectNN-C, reporting mean corruption error. Every result is reported at 128, 256 and 512 points over five seeds, alongside the artifact budget fraction (on simulated data) and the clutter fraction (on real data).
A secondary LiDAR study applies the same reliability supervision to FOGGIFY backscatter flags. Training uses a point-based detector whose first stage samples points, and evaluation uses Seeing Through Fog dense fog. The reported reference there is PointPillars at 30.7% car mAP with FOGGIFY (Hahner et al.). That number comes from a voxel detector, and no reported result was found on Lacuna for point-based detectors on that split.
Three checks decide the outcome. First, task-only samplers must over-select artifacts relative to the base rate; otherwise there is no problem to solve. Second, the reliability head must transfer to real scans, which we test by checking that its scores rank ScanObjectNN background points below object points with an area under the ranking curve clearly above chance. Third, the accuracy gain over the task-only sampler must grow as the point budget shrinks. If the gain is flat across budgets, the head is acting as a generic regularizer rather than a budget-allocation fix.
References
- Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap
- Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather
- Learnable Skeleton-Aware 3D Point Cloud Sampling
- LIP: Local Importance-Based Pooling
- Common Corruption Robustness of Point Cloud Detectors: Benchmark and Enhancement
- Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data
- Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions
- Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding