CAROM Air - Vehicle Localization and Traffic Scene Reconstruction from Aerial Videos

Duo Lu, Eric Eaton, Matt Weg, Wei Wang, Steven Como, Jeffrey Wishart, Hongbin Yu, Yezhou Yang

TL;DR

CAROM Air is a novel framework that transforms consumer-grade drone footage into high-fidelity 3D traffic reconstructions with decimeter-level localization accuracy (0.1m–0.3m). By fitting 3D vehicle "shape priors" to 2D image keypoints, the system enables low-cost, large-scale traffic safety analysis and provides a massive new dataset of 100 hours of aerial video across 50 diverse scenes.

What Problem Does This Paper Address?

Understanding how vehicles move and interact is critical for city planners, safety regulators, and autonomous vehicle (AV) developers. Traditionally, this data is collected via fixed infrastructure—cameras and sensors mounted on poles or bridges. However, this approach is expensive, limited to specific locations, and often suffers from occlusions (where one vehicle blocks the view of another).

While drones (UAVs) offer a flexible alternative, turning shaky, angled aerial video into precise 3D world coordinates is technically daunting. Most existing methods rely on simple 2D bounding boxes or top-down views, which lose critical 3D information and struggle with varying camera angles. CAROM Air seeks to bridge this gap, providing a way to extract "ground truth" style trajectory data from the air without needing any road-side hardware.

Methods

The CAROM Air framework operates in three distinct layers:

  1. The Foundational Layer (Tracking & Localization):
    • Camera Calibration: The system uses a pinhole camera model and solves the "Perspective-n-Point" (PnP) problem to determine the drone's 3D pose. Crucially, it performs continuous recalibration by tracking ground features to account for drone drift.
  2. Keypoint Detection: Instead of just drawing boxes, the researchers use a Keypoint RCNN to detect up to 19 specific vehicle features (e.g., corners of the roof, centers of wheels).
  3. 3D Model Fitting: This is the core innovation. The team performed Principal Component Analysis (PCA) on 200 different 3D vehicle models to create a "shape prior." The system optimizes the vehicle's position, heading, and shape parameters to best match the detected 2D keypoints.
  4. State Estimation: An Extended Kalman Filter (EKF) using a kinematic bicycle model smooths the trajectories and predicts vehicle motion between frames.
  5. The Middle Layer (The Dataset): A structured repository of trajectories and lane-level semantic maps.
  6. The Top Layer (Applications): Downstream tools for safety and traffic analysis.

Datasets

The authors introduce the CAROM Air Dataset, which significantly expands on existing drone-based benchmarks like highD or inD:

  • Scale: 100 hours of aerial video.
  • Diversity: 50 different scenes including roundabouts, signalized intersections, and highway segments.
  • Annotations: Includes lane-level semantic segmentation (identifying driving lanes, sidewalks, crosswalks, and buffer areas).
  • Availability: The dataset and code are open-sourced on GitHub.

Key Insights and Ideas

  • Keypoints over Bounding Boxes: Bounding boxes are viewpoint-dependent; a box looks different if the drone tilts. Keypoints are physical landmarks on the car, allowing the system to reconstruct 3D orientation and size regardless of the drone's perspective.
  • The "Shape Prior" Advantage: By using PCA to understand the "average" shape of sedans, SUVs, and trucks, the system can accurately estimate a vehicle's 3D footprint even if some keypoints are obscured or poorly detected.
  • Decimeter Accuracy: Achieving an average localization error of 0.1m to 0.3m from a drone flying at 120 meters is a major milestone, rivaling the accuracy of much more expensive ground-based LiDAR systems.
  • Auto-Labeling for Infrastructure: One of the most practical "aha moments" is using drone data to automatically label data for roadside sensors. Manually labeling 3D bounding boxes for infrastructure cameras is incredibly tedious; CAROM Air can project its high-accuracy aerial results onto the roadside camera's view to generate training labels automatically.

Key Results

  • Localization Precision: Average error of 0.1m to 0.3m.
  • Tracking Performance: Achieved Multiple Object Tracking Accuracy (MOTA) scores between 88.5% and 99.2% across various test tracks.
  • Speed Estimation: Average speed error was remarkably low, between 0.22 m/s and 0.36 m/s.
  • Safety Metrics: The paper demonstrates the ability to automatically calculate complex safety metrics like Time-To-Collision (TTC) and Post Encroachment Time (PET), which are vital for identifying "near-miss" locations before accidents actually happen.

Limitations and Future Work

Despite its success, the authors acknowledge several hurdles:

  • Operational Constraints: Drones have limited battery life (short flight times) and are subject to strict "no-fly zone" regulations in many urban areas.
  • Visual Challenges: The detector can still struggle with specular reflections (sun glinting off cars), harsh shadows, or very dark vehicles that lack visible features.
  • Rare Vehicles: The current model is optimized for standard passenger vehicles; performance may dip for motorcycles, specialized construction equipment, or long trailers.
  • Future Directions: The team aims to improve robustness for "roadside perspective" generalization and further automate the detection of aggressive driving behaviors.

Summary updated

Abstract

Road traffic scene reconstruction from videos has been desirable by road safety regulators, city planners, researchers, and autonomous driving technology developers. However, it is expensive and unnecessary to cover every mile of the road with cameras mounted on the road infrastructure. This paper presents a method that can process aerial videos to vehicle trajectory data so that a traffic scene can be automatically reconstructed and accurately re-simulated using computers. On average, the vehicle localization error is about 0.1 m to 0.3 m using a consumer-grade drone flying at 120 meters. This project also compiles a dataset of 50 reconstructed road traffic scenes from about 100 hours of aerial videos to enable various downstream traffic analysis applications and facilitate further road traffic related research. The dataset is available at https://github.com/duolu/CAROM.
BibTeX
@inproceedings{lu2023carom,
  author = {Duo Lu and Eric Eaton and Matt Weg and Wei Wang and Steven Como and Jeffrey Wishart and Hongbin Yu and Yezhou Yang},
  title = {{CAROM Air - Vehicle Localization and Traffic Scene Reconstruction from Aerial Videos}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=yVexk5S2c4},
}

Loading related content...

Figure