Abstract
This paper presents a robust multi-stage framework designed to enhance object detection performance in low-light fisheye lens traffic imagery. By integrating transformer-based image restoration, night-to-day scene conversion, and an ensemble of state-of-the-art detectors, the proposed system addresses the dual challenges of extreme geometric distortion and poor visibility. The framework achieved 5th place in the 2024 AI City Challenge Track 4 with an F1 score of 0.5965, demonstrating significant improvements over baseline methods.
Introduction
Urban traffic monitoring is increasingly adopting fisheye lenses due to their ability to provide 360-degree coverage with a single camera, thereby reducing infrastructure costs. However, these "omnidirectional" cameras introduce significant technical hurdles: extreme radial distortion, peripheral blurriness, and severely degraded visibility during nighttime. Standard object detection models, typically optimized for rectilinear, well-lit imagery, often fail to generalize to these warped, low-signal environments. This work specifically targets the detection of small, distorted objects—such as cars, pedestrians, and bicycles—in low-illumination fisheye datasets where spatial features are heavily compromised.
Proposed Framework
The authors propose a comprehensive three-stage pipeline to bridge the gap between low-quality raw captures and high-accuracy detection requirements.
Stage 1: Effective Data Pre-processing
The first stage focuses on normalizing the input data. Raw fisheye images are first processed using NAFNet (Non-linear Activation Free Network), a transformer-based architecture, to perform denoising and deblurring. Following restoration, images are categorized into "night" or "other" based on their mean RGB values. Nighttime images undergo a "Night-to-Day" conversion using a Global Structure-Aware Diffusion (GSAD) model. This step effectively "relights" the scene, reducing the domain variance that downstream detection models must handle.
Stage 2: Detection Ensemble
The framework leverages an ensemble of three high-performing detection architectures: YOLOv8x, YOLOv9e, and Co-DETR (Collaborative Detection Transformer). These models are trained on the enhanced, "day-like" dataset, allowing them to learn features from a more consistent and visible domain.
Stage 3: Robust Post-processing
To further refine results, test images are upscaled by 4x using a Dual Aggregation Transformer (DAT) before inference. This super-resolution step helps recover fine details of small objects, particularly at the distorted edges of the fisheye frame. Finally, predictions from the ensemble are merged using Weighted Box Fusion (WBF) to resolve overlapping bounding boxes and improve confidence.
Experimental Setup and Datasets
The framework was evaluated using several key datasets:
- FishEye8K: A benchmark for fisheye object detection containing 8,000 labeled images.
- AI City Challenge 2024 (Track 4): The primary competition platform for validation and testing.
- LOLv2 Synthetic & REDS: Utilized for pre-training the enhancement components (GSAD and NAFNet, respectively).
Results and Analysis
The proposed framework demonstrated both quantitative and qualitative superiority. In the 2024 AI City Challenge Track 4, the system secured 5th place among 52 teams, achieving an F1 score of 0.5965. On the FishEye8K dataset, the final ensemble (Our-2) reached an mAP of 0.489, representing a 21% relative improvement over the 0.4029 baseline.
Key insights from the results include:
- Lighting Normalization: Transforming data into a high-illumination domain is more effective than attempting to train light-invariant detectors.
- Super-Resolution Utility: Applying DAT during inference is crucial for identifying small objects in low-pixel-density regions of the fisheye lens.
- Ensemble Synergy: Combining one-stage (YOLO) and transformer-based (Co-DETR) detectors via WBF provides complementary strengths that a single model cannot match.
Conclusion and Future Work
While the framework provides a significant leap in detection accuracy for challenging fisheye environments, it introduces substantial computational overhead due to the multiple transformer stages and model ensembles. Future research should focus on developing more efficient, end-to-end architectures that integrate enhancement and detection into a single network to reduce inference latency for real-time traffic monitoring applications. Additionally, addressing potential artifacts introduced during the night-to-day conversion remains a priority for improving photorealism and detection reliability.
Summary updated