UrbanTwin Sim2Real LiDAR Challenge 2026: Site-Calibrated Synthetic Data and Multi-Detector Fusion for Roadside 3D Detection

Arnav Kumar, Brijesh SoniarXiv 2026 · arXiv

TL;DR

Team arnav007 secured first place in the UrbanTwin Sim2Real LiDAR Challenge 2026 by bridging the simulation-to-reality gap through site-calibrated synthetic data. Their approach combines a multi-detector ensemble (SECOND, VoxelNeXt, and PV-RCNN) with a sophisticated inference pipeline featuring per-scan ground leveling and "pedestrian hedging" to achieve high-precision 3D detection using zero real-world labeled training data.

What Problem Does This Paper Address?

Roadside LiDAR is a cornerstone of intelligent transportation systems, but training effective 3D object detectors is bottlenecked by the high cost and labor of annotating real-world point clouds. While synthetic data offers a scalable alternative, the "Sim2Real domain gap"—discrepancies in sensor perspective, beam patterns, and object appearances between simulation and reality—often leads to poor performance on real-world hardware.

The authors tackle the UCF UrbanTwin Sim2Real LiDAR Challenge, which tasks participants with training detectors for vehicles, pedestrians, and trucks using only synthetic data. The goal is twofold: maximize detection accuracy on real roadside scans and generate synthetic scans so realistic they are statistically indistinguishable from real ones.

Methods

The authors’ winning strategy relies on two main pillars: site-specific data generation and a robust inference-time ensemble.

1. Site-Calibrated Synthetic Generation

Instead of just building a generic digital twin, the authors modeled the specific "view" of the fixed roadside sensor. By analyzing unlabeled real scans, they estimated the range and frequency of returns for 1,024 azimuth bins and elevation bins. Crucially, they modeled "no-return" beams—points where the laser hits nothing and returns to the origin—which accounted for ~31% of the real data. This calibration ensured the synthetic training data shared the same spatial distribution and noise characteristics as the target real-world environment.

2. Multi-Detector Ensemble

The team trained three distinct architectures using the OpenPCDet toolbox:

  • SECOND: A voxel-based detector using sparse 3D convolutions.
  • VoxelNeXt: A fully sparse detector that predicts boxes directly from voxel features, allowing for a finer grid suited for small objects.
  • PV-RCNN: A two-stage detector that pools features from raw points to refine proposals.

3. The Inference Pipeline

The authors introduced several "test-time" tricks that proved vital:

  • Per-Scan Ground Leveling: Real-world sensors are often tilted (approx. 6.5° in this case). The team fitted a ground plane to every individual scan and aligned it to the synthetic "level" ground before detection, then transformed the results back.
  • Multi-Height Inference: Each scan was processed at six different ground heights to account for variations in terrain and sensor calibration.
  • Pedestrian Hedging: Because pedestrians are represented by very few points, small errors in box size can cause an Intersection over Union (IoU) failure. The team generated multiple size and position variants for each pedestrian detection to "hedge" their bets and improve recall.

Datasets

  • Synthetic Training Data: A merged set of ~18,800 frames, including "canonical" scenes, site-calibrated twins, and specialized sets with resized trucks and pedestrians to match real-world distributions.
  • Real-World Data: The V2X-Real dataset. Unlabeled scans were used for site calibration, while two small "screened" sets (150 and 350 frames) with cooperative annotations were used for validation and hyperparameter tuning.
  • Evaluation: A held-out test set of 50 real scans for detection and 50 synthetic scans for realism assessment.

Key Insights and Ideas

  • Geometry Over Features: The most significant performance boost came from per-scan ground leveling, which more than doubled the mAP (from 15.8 to 37.2 on the development set). This suggests that for LiDAR Sim2Real, geometric alignment is far more critical than matching high-level point features.
  • Modeling the "Void": Realism scores (measured by Chamfer Distance and MMD) were significantly improved by accurately modeling no-return beams. Realism isn't just about where the points are, but where they aren't.
  • Class-Specific Specialists: The authors found that no single model was best for every class. By training "specialists" and using a score-weighted fusion (BEV clustering), they could leverage the strengths of different architectures (e.g., VoxelNeXt for small pedestrians).

Key Results

  • Final Challenge Score: 0.8947, ranking 1st on the leaderboard.
  • Realism Score: 0.9507, indicating highly successful domain adaptation.
  • Detection Performance ([email protected]): 75.59% for Vehicles, 44.95% for Pedestrians, and 59.50% for Trucks.
  • IoU Sensitivity: Their submission was particularly strong at the stricter IoU 0.7 threshold (mAP 0.2646), significantly outperforming the second-place entry (mAP 0.1503).

Limitations and Future Work

The authors noted that their validation set was a poor predictor for truck detection performance, largely due to a single repeated parked truck in the development data that inflated scores. Pedestrian localization also remains a challenge due to the sparsity of points at long ranges.

Future work will focus on:

  • Developing more reliable validation sets for trucks and realism.
  • Moving from hand-tuned post-processing rules to learned fusion models.
  • Testing if their site-calibration pipeline can generalize to new roadside sensors with even less real-world data.

Summary updated

Abstract

This report summarizes the work submitted by team arnav007 (submission ID 1054222) for the UrbanTwin Sim2Real LiDAR Challenge (V2X-Real track) at the IEEE BigData Cup 2026. The primary objective is to train 3D detectors for vehicles, pedestrians, and trucks using only synthetic roadside LiDAR data, and to generate synthetic scans that are realistic enough to be indistinguishable from real ones. A CARLA based pipeline is developed and calibrated to the fixed sensor site, accurately modeling the range image, including no-return beams, based on unlabelled real development scans. A combined inference detector, using multiple 3D detectors (VoxelNeXt, PV-RCNN, and SECOND) trained exclusively on synthetic data, is also proposed. Each scan is aligned to the synthetic ground plane and processed through the detectors at multiple ground heights. The resulting bounding boxes are fused using bird's-eye view (BEV) clustering, followed by class-specific post-processing to refine heading, box size, and filter low-confidence detections. The final submission achieved a combined score of 0.8947. The ablation study shows that per-scan ground leveling more than doubled the real-scan mAP, while fusing boxes across different heights and models further improved performance.
BibTeX
@misc{kumar2026urbantwin,
  author = {Arnav Kumar and Brijesh Soni},
  title = {{UrbanTwin Sim2Real LiDAR Challenge 2026: Site-Calibrated Synthetic Data and Multi-Detector Fusion for Roadside 3D Detection}},
  year = {2026},
  eprint = {2611.20431},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20431},
}
Figure