An Optimized YOLOv5 Based Approach For Real-time Vehicle Detection At Road Intersections Using Fisheye Cameras

Md. Jahin Alam, Muhammad Zubair Hasan, Md Maisoon Rahman, Najibul Haque Sarker, Shariar Azad, Tasnim Nishat Islam, Bishmoy Paul, Tanvir Anjum, Barproda Halder, Shaikh Anowarul FattaharXiv 2025 · DOI · arXiv

Abstract

Monitoring urban intersections effectively often requires wide-angle coverage, typically achieved through 360-degree fisheye cameras. However, these cameras introduce significant non-linear barrel distortion and face extreme lighting variations. This paper presents an optimized YOLOv5-based pipeline designed for real-time vehicle detection in these challenging environments. By integrating a day-night image routing classifier, a multistage transfer learning strategy, and selective model ensembling, the proposed approach achieves a 13.7% improvement in [email protected] over the baseline YOLOv5 while maintaining real-time performance.

Introduction

Urban traffic surveillance is critical for smart city infrastructure. While 360-degree fisheye cameras offer a cost-effective way to monitor entire intersections with a single device, they present unique computer vision challenges. Vehicles at the periphery of the fisheye lens undergo severe distortion, appearing significantly different from those in the center. Additionally, environmental factors such as nighttime glare, low-light noise, and harsh daytime shadows further complicate detection. Standard object detection models often struggle to generalize across these distortions and lighting extremes. This work aims to bridge this gap by optimizing the YOLOv5 architecture for robust, real-time intersection monitoring.

Methodology

The proposed system leverages the YOLOv5 architecture as its core detection engine. A key feature of YOLOv5 is the "Focus block," which spatially squeezes pixels to preserve inter-pixel relationships, aiding the network in interpreting non-linear distortions at the edges of fisheye images.

Detailed YOLOv5 backbone architecture highlighting the Focus block and CSP modules.

To address the specific challenges of fisheye surveillance, the authors implemented a comprehensive training and inference pipeline:

  1. Day-Night Separator: A lightweight CNN classifies incoming frames as "day" or "night," routing them to specialized models optimized for specific lighting conditions.
  2. Multistage Transfer Learning: To avoid the harsh gradient shift from general objects to distorted fisheye data, weights are first pre-trained on COCO, then fine-tuned on the BDD100k dashcam dataset to learn vehicle-specific features, and finally trained on the target fisheye data.
  3. Data Augmentation and Pseudo-Labeling: An external dataset of unlabeled fisheye videos was used to generate "pseudo-labels," enriching the training pool. Furthermore, "challenging" images with low mAP scores were oversampled by 10x to improve model robustness on edge cases.
  4. Selective Ensembling: Final predictions are generated by merging bounding boxes from multiple models trained on different sequential partitions of the data.
The overall training and inference pipeline, including data preparation, sequence training, and selective ensembling.

Experimental Setup

The primary evaluation was conducted on the IEEE VIP Cup 2020 Dataset, which contains approximately 25,000 daytime and 10,000 nighttime fisheye images. Supplemental datasets included COCO for initial pre-training and BDD100k for intermediate transfer learning. A custom Pseudo Dataset was also created from surveillance videos in Thailand to further diversify the training data.

Results and Discussion

The optimized pipeline significantly outperformed the baseline YOLOv5, achieving a 13.7% increase in [email protected] on the VIP Cup dataset. The Day-Night separator achieved 100% classification accuracy, ensuring reliable routing. Nighttime performance reached a peak of 0.767 [email protected] using an ensemble of models at 768x768 resolution.

Qualitative analysis showed that the ensembled model effectively resolved complex structures, such as correctly identifying articulated trams as single vehicles and detecting cars in deep shadows or at extreme lens edges. Despite these additions, the system maintains real-time inference speeds, with the core YOLOv5s model processing frames in approximately 30ms.

An interesting observation was made regarding resolution: increasing input resolution improved Average Recall (AR) but decreased Average Precision (AP). Investigation revealed this was due to the model detecting tiny, distant vehicles missed by human annotators, resulting in "false positives" that were actually correct detections.

Conclusion

This research demonstrates that a combination of specialized routing, multistage transfer learning, and selective ensembling can overcome the inherent difficulties of fisheye-based vehicle detection. By tailoring the YOLOv5 pipeline to handle specific environmental and geometric challenges, the proposed system provides a robust solution for real-time urban traffic monitoring. Future work will focus on closing the performance gap between day and night detection by incorporating more diverse nighttime pseudo-data.

Summary updated

Abstract

Real time vehicle detection is a challenging task for urban traffic surveillance. Increase in urbanization leads to increase in accidents and traffic congestion in junction areas resulting in delayed travel time. In order to solve these problems, an intelligent system utilizing automatic detection and tracking system is significant. But this becomes a challenging task at road intersection areas which require a wide range of field view. For this reason, fish eye cameras are widely used in real time vehicle detection purpose to provide large area coverage and 360 degree view at junctions. However, it introduces challenges such as light glare from vehicles and street lights, shadow, non-linear distortion, scaling issues of vehicles and proper localization of small vehicles. To overcome each of these challenges, a modified YOLOv5 object detection scheme is proposed. YOLOv5 is a deep learning oriented convolutional neural network (CNN) based object detection method. The proposed scheme for detecting vehicles in fish-eye images consists of a light-weight day-night CNN classifier so that two different solutions can be implemented to address the day-night detection issues. Furthurmore, challenging instances are upsampled in the dataset for proper localization of vehicles and later on the detection model is ensembled and trained in different combination of vehicle datasets for better generalization, detection and accuracy. For testing, a real world fisheye dataset provided by the Video and Image Processing (VIP) Cup organizer ISSD has been used which includes images from video clips of different fisheye cameras at junction of different cities during day and night time. Experimental results show that our proposed model has outperformed the YOLOv5 model on the dataset by 13.7% mAP @ 0.5.
BibTeX
@misc{alam2025optimized,
  author = {Md. Jahin Alam and Muhammad Zubair Hasan and Md Maisoon Rahman and Md Awsafur Rahman and Najibul Haque Sarker and Shariar Azad and Tasnim Nishat Islam and Bishmoy Paul and Tanvir Anjum and Barproda Halder and Shaikh Anowarul Fattah},
  title = {{An Optimized YOLOv5 Based Approach For Real-time Vehicle Detection At Road Intersections Using Fisheye Cameras}},
  year = {2025},
  eprint = {2502.04566},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.04566},
}
Figure