A 3D segmentation network trained on clean synthetic rooms makes mistakes on real scans, but not uniformly: it fails on a minority of regions such as sensor holes, clutter, and structural elements like beams and columns. If we could locate those regions reliably, we could spend correction effort only there. The usual way to locate them is the model's own confidence, and that is exactly what breaks under domain shift. A model that has never seen real sensor noise is often confidently wrong about it. Our proposal is to take the "where is it wrong" signal from somewhere else: the real RGB frames that come with every real indoor scan. These frames can be processed by a 2D segmentation model trained on billions of real photographs, which has no synthetic-to-real gap at all. When we lift that model's object masks into 3D and they disagree with the 3D network's labels, the disagreement marks a likely error. We then route only those flagged points to small correction modules, leaving the rest of the network frozen.
The idea connects three lines of work. Real 3D labeled data is scarce while 2D real-image data is abundant, so we use 2D as an outside reference rather than as a generator. Synthetic-to-real transfer for indoor segmentation is the application. Uncertainty-guided refinement is the mechanism, with one change: the uncertainty comes from a second modality that does not share the source model's blind spots.
Core Claim
We claim that disagreement between a synthetic-trained 3D segmenter and 2D masks lifted from real images detects that segmenter's errors on real scans better than its own entropy does. We further claim that gating a small set of residual correction modules on this signal improves synthetic-to-real mIoU more than either confidence-based self-training or mask-based post-hoc voting. The first claim can be tested directly against ground-truth labels and does not depend on the second. If the first claim fails, the second has no basis.
Relation to Prior Work
Three papers motivate the gap. DODA closes much of the 3D-FRONT→ScanNet gap with two heuristics: virtual scan simulation, which casts rays to mimic sensor occlusion, and cuboid mixing, which pastes chunks of real and synthetic scenes together. It applies both uniformly, uses no uncertainty, and reports persistent failures on beams and columns. ASM learns to drop synthetic LiDAR points adversarially. It reaches 22.8% mIoU on SynLiDAR→SemanticKITTI against a 55.5% supervised oracle, which shows how much error remains after noise modeling.
Two papers supply the method. CTS uses aleatoric uncertainty on 3D box corners to filter and reweight pseudo-labels. It shows that uncertainty-aware self-training gives about 5% over a plain mean-teacher setup on CARLA3D→KITTI. However, its uncertainty is predicted by the source-trained model itself, and it only reweights losses, never triggering local correction. FineSAM++ gates lightweight residual experts on a frozen 2D backbone with a predicted uncertainty map, adding 0.7M parameters. We reuse that gating-plus-experts design in 3D. The difference is that our gate is a cross-modal disagreement score rather than a head trained on the same data whose errors it is meant to catch.
One paper constrains the design. The file notes that softmax and evidential estimators become overconfident under distribution shift, and that single-pass estimators conflate sensor noise with model ignorance. That is the failure we are routing around. It also warns that ensemble-based uncertainty, as in DGRNet and HUR-MACL, may not escape it. Ensemble members trained on the same synthetic data tend to share the same biases. We therefore treat an ensemble as a baseline, not as our signal.
We deliberately do not take the more obvious path suggested by the 3D-generation literature: using score distillation from 2D diffusion models, as in DreamPolish or GCA-3D, to hallucinate missing real geometry. For a perception task, hallucinated surfaces add plausible but unverified structure, which can introduce label noise rather than remove it. Using 2D real-image priors as a check, not as a source of new geometry, avoids that.
Research Program
The source model is a sparse-convolution segmenter trained with DODA's recipe on 3D-FRONT. On each real ScanNet scene, we run a frozen class-agnostic 2D mask model (SAM, the foundation model underlying FineSAM++) on the RGB frames. We then back-project each mask into the reconstructed point cloud using the provided camera poses and depth.
For each point $p$, let $M(p)$ be the set of lifted masks that contain it across views, and let $\hat y(q)$ be the 3D model's predicted label at point $q$. The disagreement score is
$$u(p) = 1 - \frac{1}{|M(p)|}\sum_{m \in M(p)} \frac{|\{q \in m : \hat y(q) = \hat y(p)\}|}{|m|},$$
which is the average fraction of mask-mates that carry a different label. A point whose label is an outlier within the real object it belongs to scores high. Points that appear in no mask receive a separate "unobserved" flag rather than a score.
Correction modules are small residual point-convolution heads attached to the frozen backbone's last stages. A point's features are mixed into the modules in proportion to $u(p)$, following the FineSAM++ gating form. The modules are trained on unlabeled ScanNet with two signals. The first is pseudo-labels from points with low $u$ and high confidence. The second is a consistency loss that pushes high-$u$ points toward the label carried by the majority of their lifted mask, weighted by that mask's cross-view stability.
The analysis has three parts. We measure how well $u$ separates correct from incorrect points, using ground-truth labels for evaluation only. We check whether $u$ flags the beam and column failures DODA reports. Finally, we measure how the mIoU gain depends on the fraction of points routed.
Baselines
The naive reference is the source-only model trained on 3D-FRONT and evaluated on ScanNet. The file gives only a relative figure, stating that DODA is 22% above source-only. No absolute source-only number was found on Lacuna.
The strongest existing method is DODA itself, which reports 51.4% mIoU on 3D-FRONT→ScanNet (DODA). We will reproduce it with the same backbone. Every other system will start from that reproduced checkpoint, so the comparison isolates what happens after DODA.
The strongest self-training competitor is DODA followed by confidence-thresholded pseudo-label self-training. We will run it both in plain form and with the CTS-style refinement of loss reweighting by predicted uncertainty (CTS). It gets the same unlabeled target scans, the same number of training steps, and the same threshold search budget as our method. No reported result was found on Lacuna for self-training on 3D-FRONT→ScanNet.
The baseline we consider most dangerous is DODA plus mask majority voting. Here each point is simply relabeled to the majority label of its lifted SAM masks, with no learning. It uses exactly the same 2D information as our method.
Two ablations isolate the key ingredient. In the first, the same residual experts are gated by the source model's own entropy, then by Monte Carlo dropout variance, then by random gating at a matched routed fraction. In the second, the gating is kept but the experts are removed.
Beating the field means three things. Our method must exceed reproduced DODA by a margin larger than the seed-to-seed spread across three seeds. It must exceed both confidence self-training and mask voting. And the entropy-gated ablation must fall clearly short of disagreement-gated experts.
Why not just…?
Why not just use the model's own entropy or an ensemble to decide where to refine? This is the default, and the file's own list of open problems says such estimators become overconfident under domain shift. Our first experiment tests this directly by comparing the error-detection AUROC of entropy, Monte Carlo dropout, a three-member ensemble, and $u$ on ScanNet. If entropy matches $u$, the cross-modal component is unnecessary and we will say so.
Why not just do majority voting inside the lifted 2D masks and stop there? This is a real threat, because it uses the same information at no training cost. Voting fails when a mask is wrong, for example when it merges a table with the chair beside it. It also cannot affect points the masks never cover, and it never changes the network. Our learned experts can generalize to uncovered points and to scenes at test time without RGB. We will report both, including on held-out scenes evaluated without masks. If voting matches our method on covered points, the honest conclusion is that the contribution is the signal rather than the experts. We will also check whether experts trained this way still help when no images are available at test time.
Why not just fine-tune the whole backbone, or LoRA, on pseudo-labels? FineSAM++ reports that global adaptation such as AutoSAM (0.6603 Dice on DRIVE) misses fine structures that uncertainty-gated experts recover (0.8231 Dice) (FineSAM++). That is a 2D medical result, not evidence for 3D scans. We include full fine-tuning on the same pseudo-labels, with the same steps and the same pseudo-label selection by $u$, as a baseline. This separates the value of the selection signal from the value of sparse gating.
Why not just generate real-looking geometry with 2D diffusion, as text-to-3D methods do? Methods like Kiss3DGen bake lighting into textures and produce plausible but unverified surfaces. For segmentation, invented geometry is a new source of label noise. We do not test this route, and we flag that as a scope choice.
Proposed Experiments
The decisive first check is error detection. On ScanNet validation, we compute per-point AUROC and expected calibration error for predicting "the source model is wrong," using $u$ against entropy, dropout, and ensemble scores. We report the results overall and per class, with beams and columns singled out. The proposal proceeds only if $u$ wins clearly on the categories DODA names as failures.
The second check is adaptation. We report mIoU and per-class IoU on ScanNet for every system in the Baselines section, plus ScanNet→S3DIS as a real-to-real control where the domain gap is smaller. We also plot mIoU against the fraction of routed points. That curve should rise steeply and then flatten, and if it does not, the sparse-error premise is wrong.
The third check is a transfer stress test on SynLiDAR→SemanticKITTI, using the forward camera. Masks cover only part of each LiDAR sweep, so this tests the behavior of uncovered points. The reference is ASM's 22.8% mIoU, against a 15.2% source-only baseline and a 55.5% oracle (ASM).
The outcome that would most change our view is if mask voting alone matches our method on ScanNet. That result would mean real-image masks are a sufficient fix on their own, and the residual-expert machinery should be dropped.
References
- DODA: Data-Oriented Sim-to-Real Domain Adaptation for 3D Semantic Segmentation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- CTS: Sim-to-Real Unsupervised Domain Adaptation on 3D Detection
- Gate to the Vessel: Residual Experts Restore What SAM Overlooks
- DGRNet: Disagreement-Guided Refinement for Uncertainty-Aware Brain Tumor Segmentation
- HUR-MACL: High-Uncertainty Region-Guided Multi-Architecture Collaborative Learning for Head and Neck Multi-Organ Segmentation
- DreamPolish: Domain Score Distillation With Progressive Geometry Generation
- GCA-3D: Towards Generalized and Consistent Domain Adaptation of 3D Generators
- Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation