TL;DR
The TruckDrive dataset addresses a critical gap in autonomous driving by providing a massive, highway-scale multimodal dataset tailored for long-range perception up to 1,000 meters. Built specifically for heavy-duty trucks with long braking distances, it reveals that current state-of-the-art autonomous models suffer catastrophic performance drops (up to 99%) beyond 150 meters.
What Problem Does This Paper Address?
Most existing autonomous driving datasets (like nuScenes, Waymo, and KITTI) focus on urban environments where perception ranges are typically capped around 70 to 100 meters. While this is sufficient for passenger cars navigating city streets at low speeds, it is dangerously inadequate for heavy-duty commercial trucks on highways.
At highway speeds of 120 km/h, a fully loaded truck requires 150 to 200 meters to come to a complete stop. A 100-meter sensing horizon provides a mere 3 seconds of foresight—a window entirely consumed by sensing and planning latencies before braking even begins. To enable safe highway autonomy, vehicles need a perception range extending hundreds of meters ahead. This paper introduces TruckDrive, a dataset designed to shift the research focus from short-range urban driving to long-range highway scenarios.
Methods
The authors equipped a semi-truck with a specialized, high-resolution sensor suite optimized for high-speed, long-range environments:
- 7 Long-Range FMCW LiDARs: Capable of measuring range and instantaneous radial velocity up to 400 meters.
- 3 Short-Range LiDARs: To cover blind spots near the vehicle.
- 11 Surround Cameras (8MP): Featuring varying focal lengths, including a wide-baseline stereo configuration (1.57 m) for geometric depth triangulation.
- 10 4D FMCW Radars: For robust velocity and range tracking.
To handle rolling-shutter artifacts from the high-resolution cameras, the authors implemented a cross-modal synchronization strategy that anchors the LiDAR captures to the mid-exposure timestamp of the images within a strict 5-millisecond window.
Their annotation pipeline combines manual human labeling with automated refinement. Human annotators establish initial 3D cuboids and 2D boxes. Then, a kinematically constrained optimization step (using a unicycle motion model) smooths out trajectories and reduces yaw jitter. Finally, unmatched 2D camera detections are "lifted" into 3D space using multi-view geometry and ground-height constraints from accumulated LiDAR maps.
Datasets
The paper introduces the TruckDrive dataset, which consists of:
- 475,000 total synchronized samples collected over 2 years across 8 U.S. states.
- 165,000 manually labeled frames and 310,000 unlabeled frames (to support self-supervised learning).
- Long temporal sequences (15–25 seconds) capturing realistic highway behaviors like lane changes, cut-ins, and cruise control.
Compared to traditional benchmarks, TruckDrive expands the effective 3D annotation range to $\pm400$ meters and 2D annotations up to 1,000 meters.
Key Insights and Ideas
- The Long-Range Performance Gap: While performance on urban benchmarks has begun to saturate (as shown in Figure 2), current models fail to generalize to long-range regimes.
- The BEV Resolution Dilemma: Many modern driving architectures rely on Bird's-Eye-View (BEV) grid representations. Extending these grids to cover long ranges forces a tough trade-off: either increase grid size (causing quadratic memory and compute growth) or use coarser cells, which completely erases small, distant objects or critical road debris.
- Camera Downsampling Bottleneck: Due to hardware memory constraints, SOTA models often downsample high-resolution images. For long-range stereo depth estimation, this downsampling destroys pixel disparities, leading to an $8\times$ increase in Mean Absolute Error (MAE) at far distances.
Key Results
The authors benchmarked several state-of-the-art models on TruckDrive, revealing severe performance degradation as distance increased:
- 2D Object Detection: DINO achieved a solid 63.9% mAP at short range (0–50m) but plummeted to 15.3% mAP in the ultra-long-range category (250m+).
- 3D Object Detection: Camera-only methods failed completely at long distances. Far3D dropped from 35.54% mAP at short range to a negligible 0.33% mAP at long range (150–250m). LiDAR and fusion-based models (TransFusion-L and BEVFusion) performed better due to explicit 3D structural data but still saw drops of around 31% beyond 150 meters.
- 3D Multi-Object Tracking: Tracking performance was exceptionally poor across the board, with the best model (CenterPoint) managing an AMOTA of only 13.0%, largely due to high differential velocities and long tracking gaps.
Limitations and Future Work
The primary limitation lies in the architectural assumptions of current models. Because existing networks were designed for urban environments, they cannot ingest native 8MP images or maintain fine-grained BEV grids over 400+ meters without running out of memory.
The authors conclude that simply training existing models on more data will not bridge this long-range gap. Future work must focus on developing computationally efficient, range-aware architectures, sparse representation learning, and better long-horizon temporal reasoning mechanisms specifically tailored for highway-scale autonomy.
Summary updated