This paper presents a masterclass in "data archaeology" and diagnostic engineering. Participating in the UCF UrbanTwin Sim2Real LiDAR Challenge (ECCV 2026), author Aman Upganlawar demonstrates that massive performance gains in 3D object detection often come not from complex architectural changes, but from a meticulous understanding of coordinate systems, sensor artifacts, and class-specific heuristics.
TL;DR
By focusing on diagnostic fixes—such as resolving coordinate frame mismatches, implementing per-class Non-Maximum Suppression (NMS), and modeling sensor "null returns"—the author achieved a final combined score of 0.7457 using a standard PointPillars detector. The work highlights that in Sim2Real tasks, the "gap" is often as much about data formatting as it is about domain distribution.
What Problem Does This Paper Address?
The paper tackles the Sim2Real gap in V2X (Vehicle-to-Everything) roadside LiDAR detection. In this challenge, models must be trained exclusively on synthetic "digital twin" data but are evaluated on real-world data from an Ouster OS1-64 sensor.
The challenge is twofold:
- Detection: Achieving high mean Average Precision (mAP) on real-world vehicle, pedestrian, and truck classes.
- Realism: Ensuring synthetic point clouds are distributionally similar to real ones, measured by metrics like Chamfer Distance and Earth Mover’s Distance (EMD).
Methods
The author used a standard PointPillars detector from the OpenPCDet toolbox. Rather than modifying the model, the effort was channeled into "levers" in post-processing and data alignment:
- Coordinate Frame Recovery: The author discovered a mismatch where evaluation ground truth was in a global "world" frame, while test frames were in a local LiDAR frame. Applying the inverse transform was the first major breakthrough.
- Per-Class NMS: Standard NMS uses a global Intersection-over-Union (IoU) threshold. The author found that a 0.3 IoU threshold was "killing" pedestrian detections because people often stand close together. Raising the pedestrian-specific threshold to 0.7 allowed clustered detections to survive.
- Geometric Re-centring: For small objects like pedestrians, the author replaced the predicted box center with the centroid of the LiDAR points inside it to improve localization.
- Null Return Modeling: A crucial discovery was that ~31% of the real sensor's point cloud consisted of "null returns" (points at 0,0,0). By injecting these into the synthetic data, the realism score improved significantly.
- Density-Based Re-ranking: Detections were rescored based on the number of points inside the box, demoting "ghost" boxes with few points.
Datasets
- UCF UrbanTwin Sim2Real LiDAR: A synthetic digital twin sequence of a real intersection.
- V2X-Real Track: Real roadside LiDAR data in OPV2V format.
- Ouster OS1-64: The specific sensor hardware whose characteristics (like null returns) had to be modeled.
Key Insights and Ideas
- The "Exact Zero" Diagnostic: When a correctly formatted submission returns an AP of 0.0000, it is almost certainly a coordinate frame mismatch. This insight alone moved the score from 0.3520 to 0.5147.
- Global Thresholds are Dangerous: A single NMS threshold ignores the spatial reality of different classes. Pedestrians cluster; trucks don't. Correcting this per-class raised mAP by a factor of 5.7.
- Realism is in the Artifacts: High-level realism metrics (like EMD) are sensitive to low-level data format features. The "null return" discovery (encoding missing data as 0,0,0) was a "format discovery rather than a distributional one," yet it was the single largest factor in improving the realism score.
- Calibration over Learning: Systematic biases in object dimensions (e.g., synthetic trucks being slightly different sizes than real ones) were fixed via median ratio calibration on a small non-forbidden real split.
Key Results
The final system achieved a combined score of 0.7457, comprising a detection mAP of 0.4382 and a realism score of 0.9251.
The per-class breakdown (shown below) reveals that while vehicles and trucks performed well, pedestrians remain a significant challenge for roadside LiDAR, likely due to their small footprint and the resulting localization errors.
The author also documented "Negative Results"—levers that didn't work—such as anchor-free heads (which struggled with the 18m vertical spread of the scene) and global Test Time Augmentation (TTA), which actually hurt truck detection.
Limitations and Future Work
- Pedestrian AP: At 0.1893, the sim2real gap for small objects remains wide.
- Realism Ceiling: The realism score is capped by the underlying geometry of the digital twin, which post-processing cannot fully fix.
- Sample Size: The truck class only had 124 ground truth instances, leading to higher variance in those results.
- Future Suggestions: The author suggests that challenge organizers should provide validation scripts for frame conventions and document sensor-specific encodings (like null returns) to ensure a level playing field.
Summary updated