CAROM Air - Vehicle Localization and Traffic Scene Reconstruction from Aerial Videos

Duo Lu, Eric Eaton, Matt Weg, Wei Wang, Steven Como, Jeffrey Wishart, Hongbin Yu, Yezhou Yang

Abstract

Road traffic scene reconstruction from videos is a critical tool for road safety regulators, city planners, and autonomous driving developers. However, deploying fixed infrastructure cameras across every mile of roadway is both economically and logistically impractical. This paper introduces CAROM Air, a framework designed to transform aerial video from consumer-grade drones into high-fidelity vehicle trajectory data. By fitting 3D vehicle shape priors to 2D image keypoints, the system achieves a localization error of 0.1 m to 0.3 m from an altitude of 120 meters. Alongside the methodology, we present the CAROM Air dataset, comprising 50 reconstructed scenes from 100 hours of video, facilitating advanced traffic analysis and safety research.

Introduction

Understanding vehicle interactions is fundamental to improving urban mobility and safety. Traditionally, traffic data collection has relied on fixed sensors—cameras, LiDAR, or induction loops—mounted on bridges or poles. While effective, these systems are expensive to install and maintain, and they are limited by static viewpoints that often lead to occlusions in dense traffic.

Unmanned Aerial Vehicles (UAVs) offer a flexible, cost-effective alternative for wide-area traffic monitoring. However, extracting precise 3D world coordinates from moving, angled aerial footage is a significant technical challenge. Existing methods often rely on 2D bounding boxes or orthorectified top-down views, which struggle with perspective changes and lose depth information. CAROM Air addresses these challenges by providing a robust pipeline for 3D reconstruction that rivals the accuracy of ground-based systems.

Methodology: The CAROM Air Framework

The CAROM Air system is structured into three functional layers, moving from raw visual data to actionable traffic insights.

1. Foundational Layer: Tracking and Localization

The core of the system lies in its ability to map 2D pixels to 3D space through several integrated steps:

  • Camera Calibration and Pose Estimation: The system utilizes a pinhole camera model and solves the Perspective-n-Point (PnP) problem to determine the drone's 3D position and orientation. To combat drone drift caused by wind or mechanical vibration, the system performs continuous recalibration by tracking static ground features.
  • Keypoint Detection: Rather than using traditional bounding boxes, which vary based on camera angle, the framework employs a Keypoint RCNN. This model identifies up to 19 specific physical landmarks on each vehicle, such as roof corners and wheel centers.
  • 3D Shape Prior Fitting: To reconstruct the vehicle's 3D volume, the researchers applied Principal Component Analysis (PCA) to a library of 200 diverse 3D vehicle models. This creates a "shape prior" that represents the statistical average of various vehicle types (sedans, SUVs, trucks). The system optimizes the vehicle’s position, heading, and shape parameters to align the 3D model with the detected 2D keypoints.
  • State Estimation: An Extended Kalman Filter (EKF) utilizing a kinematic bicycle model is used to smooth trajectories and provide motion predictions, ensuring temporal consistency even if a vehicle is momentarily obscured.

2. Middle Layer: Data Representation

The processed trajectories are stored alongside lane-level semantic maps. These maps include critical metadata such as driving lanes, sidewalks, crosswalks, and buffer zones, providing the necessary context for behavioral analysis.

3. Top Layer: Downstream Applications

The final layer translates raw trajectory data into safety metrics and simulation-ready formats, enabling researchers to analyze "near-miss" events and traffic flow dynamics.

The CAROM Air Dataset

A major contribution of this work is the release of the CAROM Air Dataset. Surpassing previous benchmarks like highD or inD in terms of variety, the dataset includes:

  • 100 Hours of Video: Captured across 50 distinct scenes.
  • Diverse Environments: Coverage of roundabouts, signalized intersections, and complex highway segments.
  • High-Fidelity Annotations: Precise 3D trajectories paired with semantic road maps.

The dataset and associated code are available at https://github.com/duolu/CAROM.

Results and Discussion

The CAROM Air framework demonstrates exceptional precision in real-world conditions. When tested with a drone flying at 120 meters, the system achieved:

  • Localization Accuracy: An average error of 0.1m to 0.3m.
  • Tracking Reliability: Multiple Object Tracking Accuracy (MOTA) scores ranging from 88.5% to 99.2%.
  • Velocity Estimation: Speed errors were maintained between 0.22 m/s and 0.36 m/s.

Beyond raw metrics, the system enables the automatic calculation of proactive safety indicators, such as Time-To-Collision (TTC) and Post Encroachment Time (PET). These metrics allow city planners to identify hazardous road designs before accidents occur. Furthermore, the high-accuracy aerial data can be used to "auto-label" data for roadside infrastructure cameras, significantly reducing the manual labor required to train ground-based perception systems.

Limitations and Future Directions

While CAROM Air represents a significant leap forward, certain challenges remain. UAV operations are currently limited by battery life and stringent airspace regulations. From a computer vision perspective, specular reflections, deep shadows, and extremely dark vehicles can occasionally degrade keypoint detection accuracy. Additionally, while the shape prior is robust for standard passenger vehicles, further work is needed to generalize the model for motorcycles and specialized heavy machinery.

Future research will focus on improving the system's robustness against lighting variations and extending the framework to support heterogeneous traffic types in even more complex urban environments.

Summary updated

Abstract

Road traffic scene reconstruction from videos has been desirable by road safety regulators, city planners, researchers, and autonomous driving technology developers. However, it is expensive and unnecessary to cover every mile of the road with cameras mounted on the road infrastructure. This paper presents a method that can process aerial videos to vehicle trajectory data so that a traffic scene can be automatically reconstructed and accurately re-simulated using computers. On average, the vehicle localization error is about 0.1 m to 0.3 m using a consumer-grade drone flying at 120 meters. This project also compiles a dataset of 50 reconstructed road traffic scenes from about 100 hours of aerial videos to enable various downstream traffic analysis applications and facilitate further road traffic related research. The dataset is available at https://github.com/duolu/CAROM.
BibTeX
@inproceedings{lu2023carom,
  author = {Duo Lu and Eric Eaton and Matt Weg and Wei Wang and Steven Como and Jeffrey Wishart and Hongbin Yu and Yezhou Yang},
  title = {{CAROM Air - Vehicle Localization and Traffic Scene Reconstruction from Aerial Videos}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=yVexk5S2c4},
}
Figure