Abstract
In autonomous driving, the prevailing trend has shifted toward unified end-to-end neural architectures. However, these models often struggle with sub-optimization and the integration of complex map data. Traditional modular pipelines—separating detection, tracking, and forecasting—offer better interpretability and flexibility but typically suffer from compounding errors where forecasting models fail when presented with noisy perception data. Valeo4Cast addresses this by introducing a robust finetuning strategy that bridges the gap between modules. By training the forecasting component on the imperfect outputs of upstream detection and tracking modules, the system learns to mitigate localization errors and false positives. This modular approach secured first place in the Argoverse 2 End-to-end Forecasting Challenge, outperforming the runner-up by 13.3 points and demonstrating superior performance on non-linear trajectories.
Introduction
The "holy grail" of autonomous driving is a system that can take raw sensor data and accurately predict the future trajectories of all surrounding agents. While end-to-end models attempt to solve this in a single pass, they often ignore the structural benefits of modularity. The primary challenge for modular systems is that forecasting models are typically trained on "clean" ground-truth data, making them brittle when deployed with real-world, noisy detectors. Valeo4Cast proves that modularity remains superior when the forecasting module is explicitly trained to handle the "messy" reality of perception outputs.
The Valeo4Cast Pipeline
The architecture consists of three state-of-the-art components integrated into a cohesive pipeline:
- Detection: A LiDAR-only BEVFusion model, utilizing an ensemble of three variants to capture diverse features across varying distances.
- Tracking: AB3DMOT, a training-free geometric tracker enhanced with linear interpolation to resolve fragmented tracks caused by temporary occlusions.
- Forecasting: The Motion Transformer (MTR), which leverages global and local attention mechanisms to predict multi-modal future trajectories.
Robust Finetuning Strategy
The core innovation of Valeo4Cast is its training methodology. Rather than relying solely on human-annotated ground truth, the authors finetune the MTR forecaster using the actual, imperfect outputs from the detection and tracking stages.
Many-to-One Matching
During the finetuning phase, the system employs a many-to-one matching strategy. Multiple predicted tracks—which may include redundant detections or slight localization offsets—are matched to a single ground-truth trajectory. This teaches the forecaster to be resilient against common perception failures such as identity switches and false positives. By exposing the model to the specific noise profile of its upstream components, the "compounding error" problem is effectively neutralized.
Experimental Results
Valeo4Cast was evaluated on the Argoverse 2 Sensor Dataset during the 2024 End-to-end Forecasting Challenge. The results demonstrate the significant advantages of this robust modular approach:
- Overall Performance: The model achieved 63.82 mAPf (Forecasting Average Precision), securing first place and outperforming the 2024 runner-up by 13.3 points.
- Non-Linear Trajectories: On complex maneuvers such as turns and sudden lane changes, Valeo4Cast achieved 68.4 mAPf, more than doubling the performance of the next best competitor (30.4 mAPf).
- Efficiency: Unlike massive end-to-end models that require extensive compute, the Valeo4Cast finetuning process was completed in just 12 hours on a single 8x A100 GPU node.
Conclusion
Valeo4Cast demonstrates that modularity is not a relic of the past but a powerful framework for modern autonomous driving. By focusing on the interface between modules—specifically by training the forecaster on the noise of the detector—the system achieves state-of-the-art results without the computational burden or optimization difficulties of fully integrated end-to-end models. Future work will focus on incorporating camera data and replacing heuristic post-processing for static objects with learned modules.
Summary updated