Learned Two-Plane Perspective Prior based Image Resampling for Efficient Object Detection

Anurag Ghosh, N. Dinesh Reddy, Christoph Mertz, Srinivasa G. Narasimhan

Abstract

Real-time efficient perception is critical for autonomous navigation and city scale sensing. Orthogonal to architectural improvements, streaming perception approaches have exploited adaptive sampling improving real-time detection performance. In this work, we propose a learnable geometry-guided prior that incorporates rough geometry of the 3D scene (a ground plane and a plane above) to resample images for efficient object detection. This significantly improves small and far-away object detection performance while also being more efficient both in terms of latency and memory. For autonomous navigation, using the same detector and scale, our approach improves detection rate by +4.1 APs or +39% and in real-time performance by +5.3 sAPs or +63% for small objects over state-of-the-art (SOTA). For fixed traffic cameras, our approach detects small objects at image scales other methods cannot. At the same scale, our approach improves detection of small objects by 195% (+12.5 APs) over naive-downsampling and 63% (+4.2 APs) over SOTA.

Introduction

In applications like autonomous driving and city-scale sensing, detecting small, far-away objects is critical for safety and planning. However, processing high-resolution video in real-time is computationally expensive. To fit within the "latency budget" of edge devices, engineers often resort to naive downsampling, which destroys the pixel information needed to identify distant entities.

Visualizing the two-plane perspective prior across traffic cameras, self-driving, and city sensing applications. (a) Original vs (b) Warped images show how distant objects are enlarged for better detection.

While saliency-based methods try to focus on important regions, they often rely on suboptimal heuristics or fail to generalize across different camera viewpoints. This work introduces a learnable, geometry-aware resampling strategy that preserves detail where it matters most without exceeding computational limits.

Methodology

The core innovation is the Two-Plane Perspective Prior. Instead of treating an image as a flat 2D grid, the approach incorporates the rough 3D geometry of the scene.

The Two-Plane Model

Most objects of interest lie on two planes: the ground plane (cars, pedestrians) and a plane above (traffic lights, signs). Previous geometric warping methods often used only the ground plane, which caused "nearby tall objects" (like a bus adjacent to the camera) to become heavily distorted. Adding the second overhead plane acts as a counterbalance, preserving the structure of tall objects and signals.

Geometric representation of the two-plane model (a) and the corresponding 2D image warping logic (b).

Implementation

  1. Geometric Warping: Using the scene's vanishing point, the model calculates a homography that "stretches" the distant parts of these planes, effectively enlarging small, far-away objects in the warped image.
  2. Learnable Parameters: The approach is differentiable. Parameters such as plane angles and sampling extent are learned end-to-end from the detection loss, often using pseudo-labels from a high-resolution teacher model.
  3. Integration: The system is implemented as a "Warping Layer" that sits in front of standard detectors (e.g., Faster R-CNN), making it a plug-and-play solution.

Key Results

The method was evaluated across diverse datasets including Argoverse-HD, BDD100K, WALT (4K traffic cameras), and a new Commuter Bus Dataset.

  • Small Object Detection: On Argoverse-HD, the method improved the detection rate for small objects ($AP_S$) by 39% (+4.1 points) compared to state-of-the-art adaptive sampling. In real-time streaming evaluations, the gain reached 63%.
  • Fixed Infrastructure: For traffic cameras (WALT), the method improved small object detection by 195% over naive downsampling at the same scale.
  • Edge Performance: On a Jetson AGX, the system detected 42% more rare instances while running 2.2x faster than full-resolution processing.

Conclusion

By explicitly baking 3D "common sense" into image preprocessing, this approach outperforms standard CNNs that try to learn perspective implicitly. While the method currently relies on accurate vanishing point estimation and shows a performance gap when using pseudo-labels, it provides a robust framework for efficient real-time perception. Future directions include extending this geometric prior to object tracking and trajectory prediction.

Summary updated

Abstract

Real-time efficient perception is critical for autonomous navigation and city scale sensing. Orthogonal to architectural improvements, streaming perception approaches have exploited adaptive sampling improving real-time detection performance. In this work, we propose a learnable geometry-guided prior that incorporates rough geometry of the 3D scene (a ground plane and a plane above) to resample images for efficient object detection. This significantly improves small and far-away object detection performance while also being more efficient both in terms of latency and memory. For autonomous navigation, using the same detector and scale, our approach improves detection rate by +4.1 AP s or +39% and in real-time performance by +5.3 sAPs or +63% for small objects over state-of-the-art (SOTA). For fixed traffic cameras, our approach detects small objects at image scales other methods cannot. At the same scale, our approach improves detection of small objects by 195% (+12.5 AP S ) over naive-downsampling and 63% (+4.2 AP S ) over SOTA.
BibTeX
@misc{ghosh2023learned,
  author = {Anurag Ghosh and N. Dinesh Reddy and Christoph Mertz and Srinivasa G. Narasimhan},
  title = {{Learned Two-Plane Perspective Prior based Image Resampling for Efficient Object Detection}},
  year = {2023},
  eprint = {2303.14311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.14311},
}
Figure