Self-driving perception models can be trained on synthetic LiDAR scans, the 3D point clouds a laser scanner produces, rendered in a game engine with labels for free. These models break on real sensors. On the standard synthetic-to-real benchmark the best adapted model reaches 22.8% mean intersection-over-union (mIoU, the average overlap between predicted and true regions per class), while a model trained on real labels reaches 55.5%. One promising way to close this gap is optimal transport (OT): compute a soft matching, called a transport plan, between synthetic points and real points, and use it to carry labels and pull features together. The obstacle is memory. Matching two scans of roughly $10^5$ points each requires a $10^5 \times 10^5$ plan, about 40 GB in single precision. Existing methods therefore match only a few sampled centroids or small mini-batches, which throws away the scene-level structure the matching was supposed to exploit.
We propose to match entire synthetic and real scans at once by forcing the transport plan to pass through a small set of $r$ shared "anchors," so it is stored as two thin $N \times r$ factors instead of an $N \times N$ matrix. The matching cost has two parts: how similar point features are, and how similar the 3D geometric relations between points are. The geometric term can be computed exactly in linear memory, because squared Euclidean distances in 3D form a rank-5 matrix. We also give each point transport mass in proportion to the surface area it represents, not one unit per point, so that the simulator's unrealistically complete sampling cannot drive the matching. The anchors then serve as cross-domain clusters that carry synthetic labels to real points. The central empirical question is whether matching at scene scale beats matching sampled subsets when compute is held equal. If it does not, that negative result is itself useful.
Core Claim
Two things limit current transport-based adaptation for 3D scenes, and we argue they share a single fix. The first is truncation: transport is solved on subsamples, so long-range relations such as "poles are tall thin vertical structures standing beside the road surface" are invisible. The second is the sampling shortcut: synthetic scans are complete while real scans have structured dropouts, and density alone lets a model tell the domains apart. A low-rank coupling over full scans, with a fused feature-plus-geometry cost and area-proportional mass, removes the first limit by construction. It reduces the second because mass then tracks surfaces instead of samples. Our prediction is that the gains will concentrate on thin, rare, and spatially extended classes, where subsampling and density artifacts hurt most.
Relation to Prior Work
Three papers motivate the proposal by showing that relational, instance-level matching beats global alignment, but only at truncated scale. Instance Level Affinity-Based Transfer for Unsupervised Domain Adaptation builds cross-domain affinity matrices with kNN pseudo-labels and a contrastive loss. It beats global adversarial alignment, and its authors state that the affinity matrix is memory-intensive and confined to mini-batches. Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching is the closest work: it matches local kNN graphs around farthest-point-sampled centroids with Sinkhorn OT and reaches 54.9% mIoU on vKITTI→SemanticPOSS. Because its transport runs only on sampled centroids, it never sees the whole scene; we keep transport-based matching but remove the sampling. SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation documents the same memory limit within a single domain: its $N \times N$ similarity matrix forces blocks of about $10^4$ points and produces artifacts at block seams.
Two papers supply the method. Linear-Time Gromov Wasserstein Distances using Low Rank Couplings and Costs factorizes both the cost matrices and the coupling. It solves the resulting problem by mirror descent with Dykstra projections in $O(Nr)$ memory and observes that low rank acts as a clustering regularizer. It contains no task loss or labels. We extend it to a fused cost, embed it in a self-training loop, and change the marginals. Transport Clustering: Solving Low-Rank Optimal Transport via Clustering shows that low-rank OT is equivalent to a generalized k-means. This both explains why the anchors behave like clusters and constrains us: its approximation guarantee requires a full-rank registration first, which is infeasible at our scale. We therefore rely on the direct factorized solver and accept that it can get stuck in local minima.
Three papers constrain the design. Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation (ASM) shows that domain discriminators latch onto dropout patterns rather than semantics, and it learns a masking module to hide them. SynLiDAR: Learning From Synthetic LiDAR Sequential Point Cloud for Semantic Segmentation splits the gap into geometry and beam sparsity and translates scans with heavy GAN pipelines. Keypoint-Guided Optimal Transport with Applications in Heterogeneous Domain Adaptation is a warning: purely geometric Gromov-Wasserstein matching ignores semantics and reaches only 35.6% on Office-31 heterogeneous adaptation. This is why our cost fuses features with geometry instead of using geometry alone, and why we weight the feature cost with source labels. Affine Transport for Sim-to-Real Domain Adaptation avoids pairwise matrices altogether by assuming Gaussian domains. Its authors note that this fails for high-dimensional multimodal data, which is exactly the regime of per-point scene features.
Research Program
For a synthetic scan with $N_s$ labeled points and a real scan with $N_t$ unlabeled points, let $C_{ij}$ be the distance between the backbone feature of synthetic point $i$ and real point $j$. Let $A_{ik}$ and $B_{jl}$ be squared 3D distances within each scan, and let $a, b$ be the per-point masses. We solve
$$\min_{Q,R,g}\; (1-\alpha)\langle C, P\rangle + \alpha \sum_{i,j,k,l} (A_{ik}-B_{jl})^2 P_{ij}P_{kl}, \qquad P = Q\,\mathrm{diag}(1/g)\,R^\top,$$
subject to $Q\mathbf{1}=a$, $R\mathbf{1}=b$, and $Q^\top\mathbf{1}=R^\top\mathbf{1}=g$. Here $Q \in \mathbb{R}^{N_s\times r}$ and $R \in \mathbb{R}^{N_t\times r}$ are the factors, $g$ is the mass on the $r$ anchors, and $\alpha$ trades feature cost against geometric cost.
Two facts keep memory linear. First, $A$ and $B$ have rank 5, so the quadratic term is computed exactly from thin factors. Second, $C$ is a squared feature distance, so it factors through the feature dimension. The plan $P$ is never materialized. The masses $a$ and $b$ are set proportional to a local surface-area estimate, the inverse of kNN density, instead of being uniform.
Pseudo-labels follow from the anchors. If $Y_s$ holds the one-hot synthetic labels, each anchor receives the class distribution $\mathrm{diag}(1/g)Q^\top Y_s$. Real point $j$ receives the row-normalized $R_j$ times that distribution. We keep a pseudo-label when its top-class probability passes a threshold calibrated on held-out synthetic-to-synthetic pairs.
Training alternates between two steps. In the first, we solve the coupling with stop-gradient on the current features, warm-starting from the previous solution to reduce sensitivity to initialization. In the second, we update the backbone with three terms: source cross-entropy, cross-entropy on the confident real pseudo-labels, and a loss pulling each real point's feature toward its anchor's source-feature centroid.
To make the scan pairs comparable, we pair each real scan with synthetic scans retrieved by a global scene descriptor. We study rank values $r$ from 32 to 512. As a risk-reducing extension, we also study a version with relaxed target marginals for scenes whose content does not overlap. The lead version stays balanced, because relaxed low-rank solvers are not established in the file.
Baselines
The naive reference is source-only training. Adversarially Masking Synthetic to Mimic Real reports 15.2% mIoU on SynLiDAR→SemanticKITTI and 11.5% on SynLiDAR→nuScenes, with a supervised oracle of 55.5% on SemanticKITTI. The strongest existing method on the synthetic-to-real LiDAR task is ASM, at 22.8% mIoU on SynLiDAR→SemanticKITTI and 17.0% on SynLiDAR→nuScenes, as reported in the same paper. CoSMix, at 20.0% in that paper, is a secondary reference. We will rerun ASM with its released configuration and the same segmentation backbone, training schedule, and number of tuning trials as our method. Beating it means exceeding its rerun mIoU on both targets by more than the seed-to-seed spread over three seeds.
The most important competitor for the specific claim is graph-matching OT from Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching. Our proposal differs from it mainly in the scale of the coupling, so it is the right test. That paper reports 54.9% mIoU on vKITTI→SemanticPOSS and 37.3% on SemanticKITTI→nuScenes. We found no reported result on Lacuna for graph-matching OT on SynLiDAR→SemanticKITTI, so we will reimplement it on that task. We will give it the same GPU memory and wall-clock budget as our method and let it spend that budget on as many centroids and as large a memory bank as fit.
Three ablations isolate the key ingredients. Dropping the geometric term ($\alpha=0$) reduces the method to anchor-mediated feature matching, close to prototype clustering. Uniform masses test the area-weighting. Dense Sinkhorn on subsamples of the same point count as the low-rank anchors' inputs tests whether low rank itself, rather than scale, matters.
Why not just…?
Why not just subsample more points and run ordinary Sinkhorn? This is the most serious threat. If scene-level relations add little beyond local graphs, a well-tuned subsampled matcher will tie our method. The matched-budget scale sweep described below settles this directly. If the curves are flat beyond a few thousand points, we will report that full-scene coupling is unnecessary and the claim fails.
Why not just cluster real features with k-means and align clusters to source class prototypes? Transport Clustering shows that low-rank OT is essentially a generalized k-means, so this alternative is close to our method minus two pieces: the geometric relation term and the balanced mass constraints. Its multi-pass clustering, as in Domain Consensus Clustering for Universal Domain Adaptation, is also sensitive to the initial feature quality. The $\alpha=0$ ablation is exactly this baseline with matched anchors. If it matches the full method, the geometric term contributes nothing and we will say so.
Why not just fix the sensor shortcut with learned dropout masking, as ASM does? ASM addresses how real points are sampled but carries no cross-domain correspondence, and it still leaves a 32.7-point gap to the oracle. The two approaches are complementary, so we will also report our method combined with ASM masking. A gain over ASM alone would show that relational matching adds something beyond noise modeling.
Why not just use global adversarial alignment? The instance-affinity results on Birds-31 (86.03% against 79.14% for the base network, per ILA-DA) and the semantic blurring between classes such as poles and trees in the graph-matching paper already argue against it. We include it only through the CoSMix and ASM references.
Proposed Experiments
The main benchmarks are SynLiDAR→SemanticKITTI and SynLiDAR→nuScenes, evaluated with per-class and mean IoU. The second target uses a different beam configuration, which stresses the density-shift argument. Secondary checks are vKITTI→SemanticPOSS and S3DIS→ScanNet. On these we compare against the graph-matching paper's reported 54.9% and 53.8% mIoU, using its protocol.
The decisive experiment is the scale sweep. Holding GPU memory and wall-clock time fixed, we vary the number of points entering the coupling from 1k (centroid scale) through one full scan to several accumulated scans. Subsampled dense Sinkhorn and our low-rank solver are run side by side, and we measure pseudo-label precision on real points as well as final mIoU. The claim holds only if low-rank full-scene coupling keeps improving after dense Sinkhorn runs out of memory.
The second check measures the shortcut directly. We train a small probe to predict the domain from the features of transported point pairs and compare its accuracy under area-proportional versus uniform masses. We expect area weighting to lower probe accuracy and to raise IoU most on thin and far-range classes.
The third check sweeps the rank $r$ and reports variance across random restarts. If results swing widely between restarts, it would confirm the initialization sensitivity the low-rank Gromov-Wasserstein paper warns about, and it would set how much warm-starting matters. We will also report peak memory and time per iteration, to verify linear scaling at the point counts where a dense plan would need tens of gigabytes.
References
- Instance Level Affinity-Based Transfer for Unsupervised Domain Adaptation
- Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching
- SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation
- Linear-Time Gromov Wasserstein Distances using Low Rank Couplings and Costs
- Transport Clustering: Solving Low-Rank Optimal Transport via Clustering
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- SynLiDAR: Learning From Synthetic LiDAR Sequential Point Cloud for Semantic Segmentation
- Keypoint-Guided Optimal Transport with Applications in Heterogeneous Domain Adaptation
- Affine Transport for Sim-to-Real Domain Adaptation
- Domain Consensus Clustering for Universal Domain Adaptation