TruckDrive: Long-Range Autonomous Highway Driving Dataset

Filippo Ghilotti, Edoardo Palladin, Samuel Brucker, Adam Sigal, Mario Bijelic, Felix Heide

TL;DR

The TruckDrive dataset addresses a critical gap in autonomous driving by providing a massive, highway-scale multimodal dataset tailored for long-range perception up to 1,000 meters. Built specifically for heavy-duty trucks with long braking distances, it reveals that current state-of-the-art autonomous models suffer catastrophic performance drops (up to 99%) beyond 150 meters.

What Problem Does This Paper Address?

Most existing autonomous driving datasets (like nuScenes, Waymo, and KITTI) focus on urban environments where perception ranges are typically capped around 70 to 100 meters. While this is sufficient for passenger cars navigating city streets at low speeds, it is dangerously inadequate for heavy-duty commercial trucks on highways.

At highway speeds of 120 km/h, a fully loaded truck requires 150 to 200 meters to come to a complete stop. A 100-meter sensing horizon provides a mere 3 seconds of foresight—a window entirely consumed by sensing and planning latencies before braking even begins. To enable safe highway autonomy, vehicles need a perception range extending hundreds of meters ahead. This paper introduces TruckDrive, a dataset designed to shift the research focus from short-range urban driving to long-range highway scenarios.

Figure 1. TruckDrive Dataset. Autonomous vehicles, especially heavy trucks, require long planning horizons for safe driving in highway scenarios due to higher speed and longer breaking distances. This requires perception ranges well beyond 300 m, while the most common datasets are limited to 100 m [6, 53]. We introduce the TruckDrive Dataset, a large scale multi-modal benchmark captured with a sensor setup tailored for long-range perception with LiDAR, radar and 3D annotations up to 400 m and images and 2D annotations up to 1000 m.

Methods

The authors equipped a semi-truck with a specialized, high-resolution sensor suite optimized for high-speed, long-range environments:

  • 7 Long-Range FMCW LiDARs: Capable of measuring range and instantaneous radial velocity up to 400 meters.
  • 3 Short-Range LiDARs: To cover blind spots near the vehicle.
  • 11 Surround Cameras (8MP): Featuring varying focal lengths, including a wide-baseline stereo configuration (1.57 m) for geometric depth triangulation.
  • 10 4D FMCW Radars: For robust velocity and range tracking.

To handle rolling-shutter artifacts from the high-resolution cameras, the authors implemented a cross-modal synchronization strategy that anchors the LiDAR captures to the mid-exposure timestamp of the images within a strict 5-millisecond window.

Their annotation pipeline combines manual human labeling with automated refinement. Human annotators establish initial 3D cuboids and 2D boxes. Then, a kinematically constrained optimization step (using a unicycle motion model) smooths out trajectories and reduces yaw jitter. Finally, unmatched 2D camera detections are "lifted" into 3D space using multi-view geometry and ground-height constraints from accumulated LiDAR maps.

Datasets

The paper introduces the TruckDrive dataset, which consists of:

  • 475,000 total synchronized samples collected over 2 years across 8 U.S. states.
  • 165,000 manually labeled frames and 310,000 unlabeled frames (to support self-supervised learning).
  • Long temporal sequences (15–25 seconds) capturing realistic highway behaviors like lane changes, cut-ins, and cruise control.

Compared to traditional benchmarks, TruckDrive expands the effective 3D annotation range to $\pm400$ meters and 2D annotations up to 1,000 meters.

Key Insights and Ideas

  • The Long-Range Performance Gap: While performance on urban benchmarks has begun to saturate (as shown in Figure 2), current models fail to generalize to long-range regimes.
  • The BEV Resolution Dilemma: Many modern driving architectures rely on Bird's-Eye-View (BEV) grid representations. Extending these grids to cover long ranges forces a tough trade-off: either increase grid size (causing quadratic memory and compute growth) or use coarser cells, which completely erases small, distant objects or critical road debris.
  • Camera Downsampling Bottleneck: Due to hardware memory constraints, SOTA models often downsample high-resolution images. For long-range stereo depth estimation, this downsampling destroys pixel disparities, leading to an $8\times$ increase in Mean Absolute Error (MAE) at far distances.
Performance on traditional urban datasets has saturated, masking a critical failure mode in long-range generalization.

Key Results

The authors benchmarked several state-of-the-art models on TruckDrive, revealing severe performance degradation as distance increased:

  • 2D Object Detection: DINO achieved a solid 63.9% mAP at short range (0–50m) but plummeted to 15.3% mAP in the ultra-long-range category (250m+).
  • 3D Object Detection: Camera-only methods failed completely at long distances. Far3D dropped from 35.54% mAP at short range to a negligible 0.33% mAP at long range (150–250m). LiDAR and fusion-based models (TransFusion-L and BEVFusion) performed better due to explicit 3D structural data but still saw drops of around 31% beyond 150 meters.
  • 3D Multi-Object Tracking: Tracking performance was exceptionally poor across the board, with the best model (CenterPoint) managing an AMOTA of only 13.0%, largely due to high differential velocities and long tracking gaps.

Limitations and Future Work

The primary limitation lies in the architectural assumptions of current models. Because existing networks were designed for urban environments, they cannot ingest native 8MP images or maintain fine-grained BEV grids over 400+ meters without running out of memory.

The authors conclude that simply training existing models on more data will not bridge this long-range gap. Future work must focus on developing computationally efficient, range-aware architectures, sparse representation learning, and better long-horizon temporal reasoning mechanisms specifically tailored for highway-scale autonomy.

Summary updated

Abstract

Safe highway autonomy for heavy trucks remains an open and unsolved challenge: due to long braking distances, scene understanding of hundreds of meters is required for anticipatory planning and to allow safe braking margins. However, existing driving datasets primarily cover urban scenes, with perception effectively limited to short ranges of only up to 100 meters. To address this gap, we introduce TruckDrive, a highway-scale multimodal driving dataset, captured with a sensor suite purpose-built for long range sensing: seven long-range FMCW LiDARs measuring range and radial velocity, three high-resolution short-range LiDARs, eleven 8MP surround cameras with varying focal lengths and ten 4D FMCW radars. The dataset offers 475 thousands samples with 165 thousands densely annotated frames for driving perception benchmarking up to 1,000 meters for 2D detection and 400 meters for 3D detection, depth estimation, tracking, planning and end to end driving over 20 seconds sequences at highway speeds. We find that state-of-the-art autonomous driving models do not generalize to ranges beyond 150 meters, with drops between 31% and 99% in 3D perception tasks, exposing a systematic long-range gap that current architectures and training signals cannot close.
BibTeX
@inproceedings{ghilotti2026truckdrive,
  author = {Filippo Ghilotti and Edoardo Palladin and Samuel Brucker and Adam Sigal and Mario Bijelic and Felix Heide},
  title = {{TruckDrive: Long-Range Autonomous Highway Driving Dataset}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=DsZ5XmdU0y},
}
Figure