The Earth Ain't Flat: Monocular Reconstruction of Vehicles on Steep and Graded Roads from a Moving Camera

Junaid Ahmed Ansari, Sarthak Sharma, Anshuman Majumdar, J. Krishna Murthy, K. Madhava Krishna

TL;DR

This paper tackles the critical problem of accurately locating and reconstructing the 3D shape of vehicles from a single camera, especially on challenging, non-flat roads. They achieve this by jointly estimating the vehicle's pose and shape along with the local road surface it's on, significantly outperforming previous methods that assumed flat roads.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurately perceiving their surroundings, including the precise 3D location and shape of other vehicles. While many systems use multiple sensors like cameras and LiDAR, monocular (single camera) approaches are attractive due to their lower cost and simpler integration. However, existing monocular methods for vehicle localization and shape estimation have a major limitation: they assume the road surface is flat and coplanar with the camera. This assumption breaks down in real-world scenarios with slopes, hills, and banked turns, leading to significant errors in localization and shape estimation. This paper aims to overcome this limitation by developing a monocular system that can handle arbitrarily oriented road surfaces.

Methods

The core of the proposed method is a novel optimization framework that jointly estimates several parameters:

  • Vehicle Pose and Shape: Similar to prior work, they use a learned shape prior model (a 3D wireframe representation) and estimate deformation coefficients to represent specific vehicle instances.
  • Local Road Plane: Instead of assuming a flat road, they explicitly model the local road surface beneath each vehicle as a planar patch, defined by its normal vector and distance from the origin.
  • Joint Optimization: They formulate a cost function that minimizes reprojection errors (how well the 3D model projects onto the 2D image) while simultaneously enforcing constraints related to the estimated ground plane. This includes:
  • Ground Plane Estimation Loss: Ensures the vehicle is "on" the estimated ground plane.
  • Normal Alignment Loss: Aligns the vehicle's normal (e.g., the plane of its wheels) with the estimated ground plane normal.
  • Base Point Priors: Encourages keypoints at the base of the car to lie on the estimated ground plane.
  • Global Consistency: Encourages adjacent vehicles to have consistent ground plane estimates.
  • Dimension Regularizers: Priors on vehicle dimensions to improve stability.

To initialize this process, they use a pre-trained CNN for 2D keypoint localization on vehicles and semantic segmentation to identify road areas. They also address the inherent scale ambiguity of monocular systems by leveraging known lane marking dimensions or a 3-view reconstruction approach.

Datasets

The authors evaluate their approach on two popular benchmarks:

  • KITTI Tracking Dataset: This dataset provides real-world driving sequences with varying road conditions, including some slopes and banks.
  • SYNTHIA-SF Benchmark: This synthetic dataset is specifically designed to include extremely steep roads and diverse non-planar road profiles, making it ideal for testing the robustness of their method.

Key Insights and Ideas

  • The "Flat Road" Assumption is a Major Bottleneck: The paper strongly demonstrates that the co-planarity assumption between the ego vehicle and the target vehicle is a critical failure point for existing monocular localization methods on non-flat terrain.
  • Jointly Estimating the Ground Plane is Crucial: Explicitly estimating the local road plane alongside the vehicle's pose and shape is the key innovation that allows for accurate localization on slopes.
  • Generalization from Synthetic to Real Data: Their keypoint localization network, trained entirely on synthetic data, performs well on real-world KITTI sequences without fine-tuning, highlighting the effectiveness of their synthetic data generation and training pipeline.
  • Improved Accuracy on Challenging Scenarios: The proposed method shows dramatic improvements, especially on sequences with significant slopes and banks, where previous methods fail catastrophically.

Key Results

  • KITTI Dataset: Achieved a mean localization error of 0.86 meters, a significant improvement over the previous state-of-the-art of 2.61 meters (Murthy et al. [2]). For challenging road profiles, their method achieved 0.67 meters compared to 2.55 meters for [1].
  • SYNTHIA-SF Dataset: Demonstrated a mean localization error of 0.92 meters, showcasing its ability to handle extremely steep and varied road surfaces where other methods failed drastically (e.g., a mean error of 76.34 meters for Murthy et al. [2]).
  • Qualitative Results: Figures in the paper visually confirm accurate 3D reconstruction and localization of vehicles on steep slopes, even when overlayed on ground truth data.

Limitations and Future Work

  • Limitations:
    • The method still relies on the assumption that the local road surface beneath a vehicle can be approximated by a planar patch.
  • Performance might degrade in very heavy traffic scenarios where road surfaces are not clearly visible.
  • Future Work:
    • Building denser models of roads.
  • Extending the approach to handle heavy traffic situations.

Summary updated

Abstract

Accurate localization of other traffic participants is a vital task in autonomous driving systems. State-of-the-art systems employ a combination of sensing modalities such as RGB cameras and LiDARs for localizing traffic participants, but monocular localization demonstrations have been confined to plain roads. We demonstrate - to the best of our knowledge - the first results for monocular object localization and shape estimation on surfaces that are non-coplanar with the moving ego vehicle mounted with a monocular camera. We approximate road surfaces by local planar patches and use semantic cues from vehicles in the scene to initialize a local bundle-adjustment like procedure that simultaneously estimates the 3D pose and shape of the vehicles, and the orientation of the local ground plane on which the vehicle stands. We also demonstrate that our approach transfers from synthetic to real data, without any hyperparameter-/fine-tuning. We evaluate the proposed approach on the KITTI and SYNTHIA-SF benchmarks, for a variety of road plane configurations. The proposed approach significantly improves the state-of-the-art for monocular object localization on arbitrarily-shaped roads.
BibTeX
@inproceedings{ansari2018earth,
  author = {Junaid Ahmed Ansari and Sarthak Sharma and Anshuman Majumdar and J. Krishna Murthy and K. Madhava Krishna},
  title = {{The Earth Ain't Flat: Monocular Reconstruction of Vehicles on Steep and Graded Roads from a Moving Camera}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=Uqp6INmo6g},
}
Figure