Abstract
Road traffic scene reconstruction from videos is a critical tool for road safety regulators, city planners, and autonomous driving developers. However, deploying fixed infrastructure cameras across every mile of roadway is both economically and logistically impractical. This paper introduces CAROM Air, a framework designed to transform aerial video from consumer-grade drones into high-fidelity vehicle trajectory data. By fitting 3D vehicle shape priors to 2D image keypoints, the system achieves a localization error of 0.1 m to 0.3 m from an altitude of 120 meters. Alongside the methodology, we present the CAROM Air dataset, comprising 50 reconstructed scenes from 100 hours of video, facilitating advanced traffic analysis and safety research.
Introduction
Understanding vehicle interactions is fundamental to improving urban mobility and safety. Traditionally, traffic data collection has relied on fixed sensors—cameras, LiDAR, or induction loops—mounted on bridges or poles. While effective, these systems are expensive to install and maintain, and they are limited by static viewpoints that often lead to occlusions in dense traffic.
Unmanned Aerial Vehicles (UAVs) offer a flexible, cost-effective alternative for wide-area traffic monitoring. However, extracting precise 3D world coordinates from moving, angled aerial footage is a significant technical challenge. Existing methods often rely on 2D bounding boxes or orthorectified top-down views, which struggle with perspective changes and lose depth information. CAROM Air addresses these challenges by providing a robust pipeline for 3D reconstruction that rivals the accuracy of ground-based systems.
Methodology: The CAROM Air Framework
The CAROM Air system is structured into three functional layers, moving from raw visual data to actionable traffic insights.
1. Foundational Layer: Tracking and Localization
The core of the system lies in its ability to map 2D pixels to 3D space through several integrated steps:
- Camera Calibration and Pose Estimation: The system utilizes a pinhole camera model and solves the Perspective-n-Point (PnP) problem to determine the drone's 3D position and orientation. To combat drone drift caused by wind or mechanical vibration, the system performs continuous recalibration by tracking static ground features.
- Keypoint Detection: Rather than using traditional bounding boxes, which vary based on camera angle, the framework employs a Keypoint RCNN. This model identifies up to 19 specific physical landmarks on each vehicle, such as roof corners and wheel centers.
- 3D Shape Prior Fitting: To reconstruct the vehicle's 3D volume, the researchers applied Principal Component Analysis (PCA) to a library of 200 diverse 3D vehicle models. This creates a "shape prior" that represents the statistical average of various vehicle types (sedans, SUVs, trucks). The system optimizes the vehicle’s position, heading, and shape parameters to align the 3D model with the detected 2D keypoints.
- State Estimation: An Extended Kalman Filter (EKF) utilizing a kinematic bicycle model is used to smooth trajectories and provide motion predictions, ensuring temporal consistency even if a vehicle is momentarily obscured.
2. Middle Layer: Data Representation
The processed trajectories are stored alongside lane-level semantic maps. These maps include critical metadata such as driving lanes, sidewalks, crosswalks, and buffer zones, providing the necessary context for behavioral analysis.
3. Top Layer: Downstream Applications
The final layer translates raw trajectory data into safety metrics and simulation-ready formats, enabling researchers to analyze "near-miss" events and traffic flow dynamics.
The CAROM Air Dataset
A major contribution of this work is the release of the CAROM Air Dataset. Surpassing previous benchmarks like highD or inD in terms of variety, the dataset includes:
- 100 Hours of Video: Captured across 50 distinct scenes.
- Diverse Environments: Coverage of roundabouts, signalized intersections, and complex highway segments.
- High-Fidelity Annotations: Precise 3D trajectories paired with semantic road maps.
The dataset and associated code are available at https://github.com/duolu/CAROM.
Results and Discussion
The CAROM Air framework demonstrates exceptional precision in real-world conditions. When tested with a drone flying at 120 meters, the system achieved:
- Localization Accuracy: An average error of 0.1m to 0.3m.
- Tracking Reliability: Multiple Object Tracking Accuracy (MOTA) scores ranging from 88.5% to 99.2%.
- Velocity Estimation: Speed errors were maintained between 0.22 m/s and 0.36 m/s.
Beyond raw metrics, the system enables the automatic calculation of proactive safety indicators, such as Time-To-Collision (TTC) and Post Encroachment Time (PET). These metrics allow city planners to identify hazardous road designs before accidents occur. Furthermore, the high-accuracy aerial data can be used to "auto-label" data for roadside infrastructure cameras, significantly reducing the manual labor required to train ground-based perception systems.
Limitations and Future Directions
While CAROM Air represents a significant leap forward, certain challenges remain. UAV operations are currently limited by battery life and stringent airspace regulations. From a computer vision perspective, specular reflections, deep shadows, and extremely dark vehicles can occasionally degrade keypoint detection accuracy. Additionally, while the shape prior is robust for standard passenger vehicles, further work is needed to generalize the model for motorcycles and specialized heavy machinery.
Future research will focus on improving the system's robustness against lighting variations and extending the framework to support heterogeneous traffic types in even more complex urban environments.
Summary updated