Hash-Bucket Occupancy Matching for Selecting Synthetic Training Data

Simulators and game engines can produce nearly unlimited labeled driving images and LiDAR scans. Models trained on that data still fail on real sensors, and adding more of it can make things worse. On SVHN, naively adding generated samples dropped accuracy from 92.0% to 88.1% (Generating High Fidelity Synthetic Data via Coreset selection and Entropic Regularization). This note proposes a cheap, label-free way to decide which synthetic samples to train on.

The method hashes synthetic samples and a modest pool of unlabeled real samples into the same buckets. It uses locality-sensitive hashing (LSH), a family of random projections under which nearby feature vectors tend to land in the same bucket. We then resample the synthetic stream so that its bucket occupancy matches the real data's. Synthetic samples in buckets that real data never visits are dropped. Buckets that real data fills heavily receive proportionally many synthetic samples. The selected set is chosen to look like the real distribution, rather than simply to look as "real" as possible.

Core Claim

We claim two things. The first is a claim about the selection rule, and it holds regardless of how buckets are built. Matching the real data's distribution over a coarse partition of feature space gives better sim-to-real transfer than two common alternatives. One alternative ranks synthetic samples by an estimated realness score and keeps the top ones. The other keeps a diverse subset that ignores the target altogether. The reason is that ranking by realness concentrates the budget on a few easy, real-looking modes, while target-blind diversity keeps simulator-only content.

The second claim concerns how the partition must be built for the matching to work. Hashing must happen in a content subspace, not in raw foundation-model features. The hash thresholds must also be calibrated on the real data. Without these two choices, buckets split on rendering texture instead of scene content, and occupancy is badly unbalanced.

Concretely, let $q_b$ be the fraction of unlabeled real samples in bucket $b$, and let $s_b$ be the fraction of the synthetic pool in bucket $b$. For a selection budget of $K$ samples, the method draws, without replacement,

$$ K_b = \min\left(K\,\bar q_b,\; |S_b|\right) $$

synthetic samples from bucket $b$. Here $S_b$ is the set of synthetic samples in the bucket, and $\bar q_b$ is $q_b$ averaged over several independent hash tables to smooth out bucket boundaries. When $s_b$ is small, the $|S_b|$ cap binds and leaves a shortfall. That shortfall is redistributed to the other buckets in proportion to their $\bar q_b$. The capped buckets are also reported as a coverage-hole map: regions of real scenes that the simulator does not render.

How This Differs From the Closest Work

Two LSH papers supply the machinery, and neither considers two domains. Balanced Locality-Sensitive Hashing for Online Data Selection projects pretrained embeddings onto random hyperplanes. It sets data-adaptive thresholds (mean, median, or singular value decomposition) so buckets are balanced, then draws round-robin across buckets at about 1.2 ms per step. Its goal is uniform coverage of a single training stream. We reuse the thresholding idea but calibrate it on the target domain. We also replace uniform-per-bucket sampling with target-proportional quotas. HaCore builds coresets from LSH signatures using inverted indices and bit-counting k-medoids. It assumes all data come from one distribution.

The sim-to-real work falls into three roles. Two papers motivate the target-aware approach. D²ADA shows that real samples with high target density and low source density are the most valuable ones to label. It selects real images for human annotation, however, not synthetic images for training, and it has no sublinear index. The SVHN coreset paper shows that filtering synthetic data by gradient matching (CRAIG) prevents poisoning. The authors themselves note that gradient matching is expensive.

One paper constrains the design. Driving with DINO finds that the minor principal components of DINOv3 features carry simulator texture artifacts, while the major components carry layout. Hashing raw DINO features would therefore likely bucket samples by renderer style. We hash only in the principal subspace, and we test this assumption directly rather than taking it on faith.

Three methods are complementary rather than competing, because each changes how training is done, not which data are trained on. PixMatch applies consistency training on unlabeled real images. Adversarial spatial masking learns to drop points from synthetic LiDAR scans. The SynLiDAR translator translates synthetic scans toward real appearance and sparsity. Selection composes with all three.

Research Program

We would build a streaming selector with four steps.

The first step extracts a frozen descriptor for each sample. For images, this is DINO features projected onto the top principal components computed on the real pool. For LiDAR scans, it is pooled features from a source-only segmentation encoder, concatenated with simple per-scan statistics: range histograms and point-density profiles.

The second step hashes each descriptor with $T$ tables of $B$ random hyperplanes. Each hyperplane's threshold is set to the median of the real pool's projections, so real occupancy is roughly balanced by construction.

The third step accumulates the real occupancies $q_b$ once. It then streams the synthetic pool, maintaining per-bucket reservoirs sized by $K_b$.

The fourth step trains downstream models on the selected subset, either with plain supervised training or with an existing adaptation method on top.

The analysis has three parts. First, we sweep $B$ from 8 to 20 bits to trace a bias–variance curve. Fewer bits give coarse matching; more bits give sparse, noisy occupancy estimates. We expect a sweet spot near where most real buckets hold a few dozen samples. Second, we measure bucket purity with respect to rendering versus content. This checks whether principal-subspace hashing really groups samples by scene content. Third, we examine whether the coverage-hole map predicts the classes on which the adapted model's per-class IoU (intersection over union) stays low.

Baselines

The naive reference point is uniform random subsampling of the synthetic pool at the same budget, trained identically. We also report training on the full synthetic pool. Driving in the Matrix found that scaling synthetic data keeps helping up to a diversity threshold, so full-pool training is a real bar that selection has to meet or beat.

The strongest competitor for the selection rule is density-ratio selection with a domain classifier. A logistic regression trained on the same frozen descriptors separates real from synthetic, and each synthetic sample is scored by its predicted odds of being real. We run this classifier in two modes. In top-$K$ mode it keeps the highest-scoring samples. In quota mode it uses the same bucket-quota resampling as our method, with the classifier's ratio replacing the bucket histogram. This baseline uses the identical descriptors, real pool, budget, and downstream training recipe. Its regularization gets the same tuning budget as our bit count and number of tables.

A second strong competitor is target-fit k-means occupancy matching. We cluster the real pool with k equal to the number of occupied LSH buckets, assign synthetic samples to their nearest centroid, and apply the same quotas. We also include target-blind Balanced LSH, meaning the original method run on the synthetic pool alone, and CRAIG gradient-matching coresets, the latter only on the smaller settings where they are affordable.

Three ablations isolate the proposal's ingredients. The first hashes raw DINO features instead of the principal subspace. The second uses zero thresholds or synthetic-calibrated thresholds instead of real-calibrated ones. The third replaces target-proportional quotas with uniform per-bucket quotas.

The file reports no selection results on GTA5, SYNTHIA, or SynLiDAR. The relevant reported numbers are therefore the full-data adaptation results that selection would be layered on. On GTA5 to Cityscapes, PixMatch reports 50.3 mIoU (mean IoU), and 54.5 mIoU on SYNTHIA to Cityscapes over 13 classes. On SynLiDAR to SemanticKITTI, adversarial spatial masking reports 22.8% mIoU, against 15.2% source-only and 20.0% for CoSMix. On nuScenes it reports 17.0%, against 11.5% for the baseline.

Beating the baselines means two things. At a 10–25% budget, our selection plus PixMatch or plus masking should match or exceed these full-pool figures. It should also beat both the classifier baseline and uniform random selection at equal budget, by a margin larger than the spread across seeds.

Why not just…?

Why not just subsample randomly or use everything? Random subsampling keeps synthetic content in proportion to what the simulator happens to render, not to what the real world contains. The SVHN result (92.0% to 88.1% with naive addition, recovered to 94.5% with filtering) shows that uncurated synthetic data can hurt. Still, a heavy-traffic driving simulator may already roughly match real content statistics. If random selection at 25% ties our method, that is a legitimate negative finding, and the experiments report it plainly.

Why not just train a domain classifier and keep the most real-looking samples? This is the most serious threat, and it is why the classifier appears as a baseline in two modes. It is plausible that the classifier's ratio is a better estimator than a bucket histogram. Our specific hypothesis is that top-$K$ ranking collapses the budget onto easy modes, such as frames dominated by road and car, and starves rare classes. The quota-mode classifier separates the two questions. If the quota-mode classifier matches our method, the contribution reduces to "use occupancy-matched resampling, with whatever ratio estimator." In that case, LSH is justified only by streaming cost and its no-training property, and we would say so.

Why not just run k-means on the real pool? On a few thousand Cityscapes images, k-means fitting is cheap, so at image scale it is a genuine peer. LSH has three structural advantages. It needs no fitting. It gives $2^B$ buckets at no extra cost, with multiple tables that soften boundary effects. Its occupancies can be updated online if the real pool grows or drifts. The experiment compares both methods at equal bucket counts on images and on scan-level LiDAR. If k-means wins or ties, we will present the scientific result (matching beats ranking) independently of the hashing.

Why not just rely on the adaptation method? Consistency training and adversarial masking change how training uses data, not which data are used. The proposal succeeds only if selection adds to them, so every headline comparison stacks selection on top of PixMatch or masking rather than replacing them.

Proposed Experiments

We use two main settings. The image setting is GTA5 and SYNTHIA to Cityscapes semantic segmentation, reported as mIoU. The real pool is the unlabeled Cityscapes training images, and the full-pool reference numbers are PixMatch's 50.3 and 54.5. The LiDAR setting is SynLiDAR to SemanticKITTI and to nuScenes, also reported as mIoU. The real pool is unlabeled training scans, and the full-pool reference numbers are the masking method's 22.8% and 17.0%.

We sweep budgets of 5%, 10%, 25%, and 50%, with three seeds each. We also report wall-clock selection cost per sample and compare it to the ~1.2 ms per-step figure from Balanced LSH. A small sanity setting on SVHN with generated augmentations checks whether bucket-occupancy filtering alone recovers part of the gap between 88.1% and 94.5% that CRAIG closed, at a fraction of CRAIG's cost.

The decisive checks are three.

The first test is on the selection rule. Occupancy matching must beat top-$K$ classifier selection on mean IoU at equal budget. It must also show a smaller spread of per-class IoU across rare classes. If it does not, the core claim fails.

The second test concerns the partition. Principal-subspace hashing with real-calibrated thresholds must beat raw-feature hashing. The raw-feature variant should visibly cluster samples by renderer, which we measure by the fraction of buckets that are over 95% synthetic. If it does not, the Driving with DINO constraint does not transfer to selection.

The third test is additivity. Selection plus an adaptation method at 25% budget must match or exceed that method on the full pool. Failing this would mean selection only saves compute rather than improving transfer, which would still be useful but is a weaker claim.