OmniDRL: Robust Pedestrian Detection using Deep Reinforcement Learning on Omnidirectional Cameras

G. Dias Pais, Tiago J. Dias, Jacinto C. Nascimento, Pedro Miraldo

Abstract

Pedestrian detection is one of the most explored topics in computer vision and robotics. The use of deep learning methods allowed the development of new and highly competitive algorithms. Deep Reinforcement Learning has proved to be within the state-of-the-art in terms of both detection in perspective cameras and robotics applications. However, for detection in omnidirectional cameras, the literature is still scarce, mostly because of their high levels of distortion. This paper presents a novel and efficient technique for robust pedestrian detection in omnidirectional images. The proposed method uses deep Reinforcement Learning that takes advantage of the distortion in the image. By considering the 3D bounding boxes and their distorted projections into the image, our method is able to provide the pedestrian's position in the world, in contrast to the image positions provided by most state-of-the-art methods for perspective cameras. Our method avoids the need of pre-processing steps to remove the distortion, which is computationally expensive. Beyond the novel solution, our method compares favorably with the state-of-the-art methodologies that do not consider the underlying distortion for the detection task.

Introduction

Omnidirectional cameras, such as fisheye or catadioptric systems, are increasingly popular in robotics and surveillance due to their massive Field of View (FoV). However, these systems introduce significant radial distortion, causing objects to appear curved and stretched. Traditional detection methods, like Faster R-CNN, are optimized for perspective cameras and often fail or require computationally expensive undistortion pre-processing. This paper introduces OmniDRL, a framework that leverages Deep Reinforcement Learning (DRL) to detect pedestrians directly in distorted images while simultaneously recovering their 3D world positions.

Methodology

OmniDRL reformulates pedestrian detection as an iterative search problem solved by a Reinforcement Learning agent. Rather than fighting distortion, the system uses the mathematical model of the camera's geometry to its advantage.

3D Bounding Box Parametrization

Unlike standard 2D detectors that search for rectangles in image space, the OmniDRL agent manipulates a 3D bounding box defined in cylindrical coordinates: distance ($\rho$), angle ($\beta$), height ($h$), and width ($w$). This allows the system to account for the physical geometry of the environment and provides the actual 3D position of the pedestrian relative to the camera.

Environment setup showing the input image and the 3D coordinate system used for bounding box parametrization.

Multi-task Deep Q-Network (DQN)

The system uses a unified spherical camera model to project the 3D box edges onto the 2D image plane. These projections naturally follow the image's distortion. A Multi-task Deep Q-Network (DQN) architecture is employed, splitting into two branches: a DQN branch that learns a policy to adjust the 3D box through actions (e.g., "Turn Right", "Increase $\rho$"), and a Classification branch that provides a prior to initialize the search.

The iterative search process of the DRL agent, showing actions like turning, scaling, and increasing distance over time steps to reach a trigger.

Experimental Results

The authors evaluated OmniDRL on a custom dataset of 921 omnidirectional images captured in various laboratory environments. The results demonstrate significant improvements over existing methods:

  • Accuracy: OmniDRL achieved a 0.718 Intersection over Union (IoU), far exceeding the 0.385 IoU of state-of-the-art perspective-based DRL methods on distorted images.
  • Detection Rate: The system correctly detected pedestrians 86.5% of the time, whereas Faster R-CNN struggled at 28.6% on the same distorted data.
  • Efficiency: The agent was highly efficient, requiring an average of only 21.39 steps to trigger a detection, compared to over 62 steps for previous DRL-based detectors.

Conclusion

OmniDRL demonstrates that embracing image distortion rather than removing it leads to more robust and efficient pedestrian detection in 360° views. By operating in 3D world coordinates, the system provides immediate localization data essential for autonomous navigation and human-aware robotics. Future work will focus on scaling the system to multi-pedestrian scenarios and more diverse outdoor environments.

Summary updated

Abstract

Pedestrian detection is one of the most explored topics in computer vision and robotics. The use of deep learning methods allowed the development of new and highly competitive algorithms. Deep Reinforcement Learning has proved to be within the state-of-the-art in terms of both detection in perspective cameras and robotics applications. However, for detection in omnidirectional cameras, the literature is still scarce, mostly because of their high levels of distortion. This paper presents a novel and efficient technique for robust pedestrian detection in omnidirectional images. The proposed method uses deep Reinforcement Learning that takes advantage of the distortion in the image. By considering the 3D bounding boxes and their distorted projections into the image, our method is able to provide the pedestrian's position in the world, in contrast to the image positions provided by most state-of-the-art methods for perspective cameras. Our method avoids the need of pre-processing steps to remove the distortion, which is computationally expensive. Beyond the novel solution, our method compares favorably with the state-of-the-art methodologies that do not consider the underlying distortion for the detection task.
BibTeX
@inproceedings{pais2019omnidrl,
  author = {G. Dias Pais and Tiago J. Dias and Jacinto C. Nascimento and Pedro Miraldo},
  title = {{OmniDRL: Robust Pedestrian Detection using Deep Reinforcement Learning on Omnidirectional Cameras}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=u01ybLBcX1L},
}
Figure