OmniDRL: Robust Pedestrian Detection using Deep Reinforcement Learning on Omnidirectional Cameras

G. Dias Pais, Tiago J. Dias, Jacinto C. Nascimento, Pedro Miraldo

TL;DR

OmniDRL is a novel framework that uses Deep Reinforcement Learning (DRL) to detect pedestrians directly in highly distorted omnidirectional (360°) images. By manipulating 3D bounding boxes in world coordinates rather than 2D rectangles in image space, the system achieves higher accuracy and efficiency without the need for computationally expensive "undistortion" pre-processing.

What Problem Does This Paper Address?

Omnidirectional cameras (like fisheye or catadioptric systems) are favorites in robotics and surveillance because of their massive Field of View (FoV). However, they come with a major headache: extreme radial distortion.

Standard object detection algorithms (like Faster R-CNN) are designed for perspective cameras where straight lines stay straight and objects fit neatly into rectangular boxes. In a 360° image, a pedestrian might look like a curved, stretched noodle. The traditional fix is to "undistort" the image first, but this is slow, introduces visual artifacts, and stretches objects in ways that still don't fit standard bounding boxes. The authors set out to find a way to detect pedestrians natively in these distorted views while simultaneously recovering their 3D position in the world.

Methods

The authors propose OmniDRL, which treats pedestrian detection as a search problem solved by a Reinforcement Learning agent.

  1. 3D Bounding Box Parametrization: Instead of searching for a 2D box $(x, y, w, h)$, the agent manipulates a 3D box in cylindrical coordinates: distance ($\rho$), angle ($\beta$), height ($h$), and width ($w$).
  2. Distortion-Aware Projection: Using a unified spherical camera model, the system projects the edges of this 3D box onto the 2D image plane. Because it accounts for the camera's geometry, these projections appear as curves that perfectly match the image's distortion.
  3. Multi-task Deep Q-Network (DQN): The architecture splits into two branches that share early convolutional layers:
  4. DQN Branch: Learns a policy to move and resize the 3D box (actions like "Turn Right," "Increase $\rho$," etc.) to home in on a pedestrian.
  5. Classification Branch: Acts as a "prior," helping to initialize the search and determine if a pedestrian is actually present in a proposed region.
  6. Double DQN (DDQN): To prevent the agent from being overly optimistic about its actions, they use a Double DQN approach to stabilize training.

Datasets

Because no standard benchmarks existed for pedestrian detection in omnidirectional settings at the time, the authors created their own:

  • Custom Omnidirectional Dataset: 921 images captured in three different laboratory environments using a Flea3 camera with a hyperbolic mirror.
  • Split: 70% for training, 30% for testing.
  • Ground Truth: Manually labeled with distorted bounding boxes that correspond to 3D world positions.

Key Insights and Ideas

  • Embracing the Distortion: Rather than fighting distortion by trying to remove it, OmniDRL uses the mathematical model of the distortion to its advantage. The agent "understands" how an object's shape should change as it moves through the curved space.
  • 3D Localization for Free: Because the agent operates in world coordinates ($\rho, \beta$), the output isn't just a box on a screen—it’s the actual 3D position of the pedestrian relative to the robot/camera. This is a huge win for navigation and "human-aware" robotics.
  • The Power of the Prior: The multi-task approach (combining classification with DRL) significantly reduces the number of steps the agent needs to take to find a target, making the system much faster than previous DRL-based detectors.

Key Results

The authors compared OmniDRL against state-of-the-art (SotA) methods like Caicedo’s DRL detector and Faster R-CNN, tested on both distorted and undistorted images:

  • Accuracy: OmniDRL achieved a 0.718 Intersection over Union (IoU), significantly higher than the SotA DRL method (0.385) on distorted images.
  • Detection Rate: It correctly detected pedestrians 86.5% of the time, outperforming Faster R-CNN (which struggled at 28.6% on distorted images).
  • Efficiency: The agent required an average of only 21.39 steps to trigger a detection, compared to over 62 steps for the perspective-based SotA method.
  • Robustness: The system proved much more capable of handling pedestrians near the edges of the FoV where distortion is most severe.

Limitations and Future Work

  • Single Target Assumption: The current classification branch assumes one pedestrian per image to simplify the "prior" initialization. Scaling this to crowded scenes with many overlapping pedestrians would be a logical next step.
  • Dataset Scale: While the custom dataset is a great start, 921 images is small by modern deep learning standards. Testing on more diverse, outdoor, and dynamic environments would further prove the system's robustness.
  • Generalization: While the spherical model is general, the specific experiments focused on catadioptric (mirror-based) systems. Testing on various fisheye lenses would confirm the "Omni" in OmniDRL.

Summary updated

Abstract

Pedestrian detection is one of the most explored topics in computer vision and robotics. The use of deep learning methods allowed the development of new and highly competitive algorithms. Deep Reinforcement Learning has proved to be within the state-of-the-art in terms of both detection in perspective cameras and robotics applications. However, for detection in omnidirectional cameras, the literature is still scarce, mostly because of their high levels of distortion. This paper presents a novel and efficient technique for robust pedestrian detection in omnidirectional images. The proposed method uses deep Reinforcement Learning that takes advantage of the distortion in the image. By considering the 3D bounding boxes and their distorted projections into the image, our method is able to provide the pedestrian’s position in the world, in contrast to the image positions provided by most state-of-the-art methods for perspective cameras. Our method avoids the need of pre-processing steps to remove the distortion, which is computationally expensive. Beyond the novel solution, our method compares favorably with the state-of-the-art methodologies that do not consider the underlying distortion for the detection task.
BibTeX
@inproceedings{pais2019omnidrl,
  author = {G. Dias Pais and Tiago J. Dias and Jacinto C. Nascimento and Pedro Miraldo},
  title = {{OmniDRL: Robust Pedestrian Detection using Deep Reinforcement Learning on Omnidirectional Cameras}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=u01ybLBcX1L},
}

Loading related content...

Figure