Models trained on simulated sensor data often fail on real sensors. Part of the reason is that the network learns to rely on quirks of the simulator rather than on the scene. In LiDAR (laser range scanning), a simulator returns clean ranges and intensities. A real scanner returns jittered ranges, intensities that depend on surface material, and dropped beams. Many of these differences are small, continuous shifts in signal magnitude, and early convolutional features carry them along faithfully. We propose a simple test of whether discarding magnitude helps. A shallow layer of the segmentation network is forced to output only the sign of each feature, a binary code in $\{-1,+1\}$. The layer is trained end-to-end by gradually sharpening a smooth approximation of the sign function. If the domain-specific signal lives mostly in magnitudes, a sign-only code should keep the scene's spatial pattern and drop much of what separates synthetic from real.
This brings an established technique, differentiable binarization by continuation, to a place it has not been tested. Continuation means training through $\tanh(\beta x)$ while raising the steepness $\beta$ until the function behaves like a sign. HashNet developed it for compact retrieval codes. We apply it to the synthetic-to-real LiDAR benchmark SynLiDAR→SemanticKITTI. There the best adapted model in the file reaches 22.8% mean intersection-over-union (mIoU), while a model trained on real data reaches 55.5%. A useful side effect is that the bottleneck layer becomes a bit-packed tensor that is cheap to store and move on embedded hardware.
Core Claim
We claim that binarizing the activations of one early layer reduces the synthetic-to-real gap more than other ways of throwing away information at the same place. The comparison is against continuous bottlenecks with the same bit budget and against normalization layers. The proposed mechanism is specific. Sensor discrepancies such as range noise, intensity calibration, and distance falloff mostly perturb feature magnitudes, and they rarely flip feature signs. A sign code is therefore invariant to a class of perturbations that per-channel normalization and low-dimensional projection do not remove.
The claim is narrower than "binary representations are domain-invariant," and it can fail in identifiable ways. It fails if a magnitude-removing normalization matches the gain. It also fails if a continuous code with matched bit capacity matches it. In either case, binarization would be doing no more than reducing capacity.
Relation to Prior Work
Two papers motivate the idea. Adversarially Masking Synthetic to Mimic Real (ASM) found that domain discriminators separate synthetic from real LiDAR by spotting "clean versus noisy" signatures rather than semantics. Its fix is a learned, Gumbel-Softmax binary keep/drop mask on input points, which imitates real missingness. Our proposal attacks a different part of the gap: magnitude-borne noise on the points that survive. We binarize the features, not the point set, so the two methods can be combined. Sim-to-Seg showed for off-road driving that an intermediate canonical representation strips visual style. It reduced L2 distance to goal to 0.287, compared with 0.653 for a hand-engineered classical driving stack. However, Sim-to-Seg uses a hand-chosen semantic representation. Our representation is a learned binary code inside the network.
Three papers supply the method. HashNet introduced continuation training for binary codes and documented the cost of binarizing after training: 16-bit ImageNet MAP of 0.3106 for DHN against 0.5059 with continuation. Differentiable Binarization and its adaptive-scale extension supply the learned-threshold form $\sigma(k(P-T))$. We test it as a variant in which each channel learns its own threshold rather than using zero.
Two papers constrain the design. Differentiable Weight Masks for Domain Transfer found on PACS that binary weight masks protect source knowledge but adapt poorly to the target. For that reason we binarize one activation layer only and leave all weights continuous. Why Quantization Improves Generalization shows that binary-weight networks have a smaller effective capacity and a smaller in-distribution generalization gap. Its analysis says nothing about distribution shift and uses infinite-width assumptions, so we treat it as weak motivation and not as a prediction. STUN uses "binary" to mean one-vs-all classifiers, and it is not a precedent for binary codes.
Research Program
Let $h$ be the feature map after an early block of the segmentation backbone, and let $W$ be a $1\times1$ projection to $B$ channels. During training the bottleneck outputs
$$z_t = \tanh\big(\beta_t\,(W h - \tau)\big),$$
where $\tau$ is a per-channel threshold, either fixed at zero or learned, and $\beta_t$ is a steepness that grows over training iterations $t$. At test time, $z = \mathrm{sign}(Wh - \tau)$ is fed to the rest of the network. For the schedule we follow HashNet's stepwise increase in $\beta$. Because HashNet flags schedule sensitivity as an open problem, we report results across three schedules rather than one tuned schedule.
The backbone and training recipe are whatever ASM uses, reproduced first so that its reported numbers are matched. The bottleneck is inserted at the same shallow depth where ASM places its mask, since that is where sensor-level differences enter the network. Training uses labeled SynLiDAR and unlabeled real scans. The main variant uses no adversarial loss, so we can see what binarization does by itself. A second variant adds ASM's discriminator and masking, which tests whether the two fixes for missingness and magnitude stack.
The analysis has two parts. The first is a probe of the mechanism. On a frozen trained model, we fit a small domain classifier to the bottleneck output and to the matched continuous alternatives. We also inject controlled perturbations into held-out synthetic scans, namely Gaussian range jitter, intensity rescaling, and distance-dependent attenuation, at increasing strength. We then measure the fraction of code bits that flip. The mechanism predicts few bit flips under magnitude perturbations and lower domain-classifier accuracy than any matched continuous code. It also predicts that layers with lower domain-classifier accuracy yield larger mIoU gains.
The second part is a sweep over code width $B$ and insertion depth. This addresses the trade-off between suppressing shortcuts and keeping capacity, raised by the weight-mask and quantization-NTK papers. We expect gains at moderate $B$ and losses at very small $B$. We report the full curve, not its best point.
Baselines
The naive reference is source-only training of the same backbone. ASM reports 15.2% mIoU on SemanticKITTI and 11.5% on nuScenes for its source-only model. Above that sits CoSMix, a mixing-based adaptation method, which the ASM paper reports at 20.0% on SemanticKITTI.
The strongest existing method is ASM itself, at 22.8% mIoU on SynLiDAR→SemanticKITTI and 17.0% on SynLiDAR→nuScenes, as reported in ASM. It is the right competitor because it uses a differentiable binary operation at the same depth to remove a sensor shortcut. All methods are rerun with the same backbone, data splits, and iteration count, and each gets the same number of hyperparameter trials. If our reproduction of ASM misses its reported number, we report both. Beating ASM means the binary bottleneck alone exceeds 22.8% on SemanticKITTI, or that ASM plus the bottleneck exceeds ASM rerun under our setup, across three seeds with non-overlapping spread. Matching ASM without a discriminator would also count as a meaningful result, because the method would be simpler.
Three ablations isolate the key ingredient, which is removing magnitude by taking the sign. The first is a continuous bottleneck of width $B/b$ quantized to $b$ bits per channel, which gives the same total bits and keeps some magnitude. The second is the same continuous bottleneck with no quantization, which tests capacity reduction alone. The third places instance normalization or per-channel L2 normalization at the same location, which removes scale without discretizing. The binary code must beat all three for the claim to stand.
For the training method, we compare continuation against Gumbel-Softmax and straight-through sign estimators at a matched code. This tells us whether the training procedure matters or only the final binary code does.
Why not just…?
Why not just randomize sensor noise in the simulator? Randomizing range jitter, intensity, and dropout is the cheapest competitor, and it is a real threat for the magnitude part of the gap. It needs the right noise families, though, and both the Sim-to-Seg and COMPASS papers (COMPASS) report that undirected randomization falls short in their settings. Those settings are vision-based driving and physical-parameter tuning, not LiDAR, so the evidence is indirect. We therefore run a domain-randomization baseline using the same perturbation families as the mechanism probe. We also run it combined with the bottleneck. If randomization alone matches the bottleneck, the honest conclusion is that binarization adds nothing, and we will report it that way.
Why not just normalize the features? Instance or L2 normalization also removes scale, and it is the most direct threat to the specific mechanism. The difference is that normalization removes one scale factor per channel or location, while a sign code removes magnitude independently for every element. Only the latter is invariant to element-wise noise such as range jitter. The normalization ablation in the Baselines section settles this directly.
Why not just binarize a trained continuous model after the fact? HashNet's 0.3106 versus 0.5059 MAP gap shows that rounding after training costs a lot when the task depends on the code. We include post-hoc sign thresholding of the continuous-bottleneck model to confirm that this holds for segmentation transfer as well.
Why not just tune ASM's Gumbel temperature? ASM addresses missingness, which is a different shortcut, so tuning its masking schedule cannot remove magnitude artifacts on the points that remain. The ASM-plus-bottleneck experiment tests whether the two gains are complementary. If they are not, the two methods are removing the same shortcut, and the bottleneck is redundant.
Proposed Experiments and Decisive Checks
The primary benchmark is SynLiDAR→SemanticKITTI, using SynLiDAR and SemanticKITTI. SynLiDAR→nuScenes tests transfer to a 32-beam sensor. We report per-class IoU in addition to the mean, because the mechanism predicts the largest gains on classes whose real returns have unusual intensity, such as vegetation, glass-heavy buildings, and distant cars. As an inexpensive check that the mechanism is not specific to LiDAR, we run MNIST→MNIST-M, where the gap is in background color and texture. PixelDA reports 98.2% accuracy there, so we treat this purely as a probe of the mechanism and do not compete with that number.
Three outcomes would be decisive. The first is that the binary bottleneck beats all matched-capacity continuous and normalization ablations on SemanticKITTI mIoU, across seeds. The second is that domain-classifier accuracy on the binary code is clearly lower than on those ablations, and that the bit-flip rate stays low under magnitude perturbations but not under geometric ones. The third is that adding the bottleneck to ASM improves on ASM. If the first outcome holds but the second does not, the gain is real but our explanation is wrong, and we will say so. If the normalization ablation matches the binary code, the paper's conclusion becomes that scale removal, not binarization, is what transfers.
References
- HashNet: Deep Learning to Hash by Continuation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data
- Real-Time Scene Text Detection with Differentiable Binarization
- Real-Time Scene Text Detection With Differentiable Binarization and Adaptive Scale Fusion
- Differentiable Weight Masks for Domain Transfer
- Why Quantization Improves Generalization: NTK of Binary Weight Neural Network
- Stochastic Binary Network for Universal Domain Adaptation
- What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
- SynLiDAR: Learning From Synthetic LiDAR Sequential Point Cloud for Semantic Segmentation
- SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks