TL;DR
This paper introduces a novel method for automatically correcting distortions in wide-angle and fish-eye camera images. By refining outlier line segments and using a cumulative plumbline angular loss, their approach significantly improves the accuracy of downstream vision tasks like object detection and height estimation, especially in challenging real-world scenarios.
What Problem Does This Paper Address?
Wide-angle and fish-eye lenses are crucial for capturing broad scenes in applications like autonomous driving (ADAS) and surveillance. However, these lenses introduce significant radial distortions, especially at the image edges. These distortions can severely degrade the performance of computer vision algorithms by causing pixel inconsistencies, stretching, and inaccurate geometric measurements. Existing methods often struggle with extreme distortions, varying illumination, and require manual intervention or model-specific knowledge, limiting their real-time applicability. This paper aims to provide a robust, automatic solution that handles these challenges effectively.
Methods
The core of this work lies in an outlier refinement scheme integrated with a plumbline angular loss function.
- Lens Distortion Model: They employ an odd polynomial lens distortion model with up to two coefficients ($D_1, D_2$) to map rectified pixel coordinates to distorted ones.
- Line Segment Extraction: Line segments are extracted using a parameter-free edge drawing algorithm and filtered based on length heuristics. These segments are grouped into "line-member sets" originating from the same edge source.
- Plumbline Angular Loss: This loss function measures the angular difference between line members within a set. It's designed to penalize deviations from expected straightness in the rectified image.
- Outlier Refinement: The key innovation is an iterative process that refines the line-member sets. By minimizing the cumulative loss over these sets, the algorithm simultaneously identifies and removes outlier line segments that cause significant errors. This iterative refinement leads to a more robust set of "line members."
- Optimization: The Levenberg-Marquardt (LM) algorithm is used to estimate the distortion parameters ($f_x, f_y, c_x, c_y, D_1, D_2$) by minimizing the refined cumulative loss. This process aims to achieve simultaneous outlier elimination and parameter estimation.
Datasets
The study utilizes a combination of datasets for evaluation:
- Synthetic Distorted KITTI Dataset: Synthetically modified from the original KITTI dataset using open-sourced distortion induction codes. This allows for quantitative evaluation of rectification algorithms and high-level metrics against ground truth.
- Private-Real Datasets: Collected using various cameras with different lens models (fish-eye, wide-angle, super wide-angle). These datasets are used to test the robustness of the algorithms in real-world scenarios, particularly for object detection and height estimation.
- Computer Vision Lab, Inha University Dataset: Used for height estimation experiments, including data collected at night and under severe illumination changes.
Key Insights and Ideas
- Cumulative Loss for Robustness: Aggregating loss over sets of line members, rather than individual segments, provides a more stable and robust measure, especially when dealing with noisy or extreme distortions.
- Iterative Outlier Refinement is Crucial: Simply using a straightness constraint isn't enough. The iterative removal of outlier line segments is essential for achieving accurate rectification, particularly in challenging scenarios with severe distortions.
- Streamlining Vision Tasks: The primary goal is not just distortion rectification but enabling better performance in downstream tasks. The paper demonstrates that accurate rectification directly translates to improved object detection and height estimation.
- Model-Agnostic Approach: The method aims to be flexible and handle various lens models without requiring prior model-specific knowledge or empirical factors, which was a limitation in previous work.
Key Results
- Image Quality: The proposed method achieved high scores in PSNR (19 dB), SSIM (47.2%), S3 (0.51), and LPC-SI (0.92) on synthetic distorted datasets, outperforming traditional methods and prior work.
- Pixel-Point Error: The average pixel-point error was significantly reduced to 18.3 pixels, compared to 24.1 (Santana et al.) and 29.2 (Alvarez et al.) on synthetic distortion center benchmarks.
- Object Detection (ADAS): On rectified KITTI samples, the proposed method achieved higher mAP scores (72.4% for SSD, 79.8% for YOLOv3) compared to other rectification algorithms, demonstrating improved performance for object detectors. Qualitative results showed better detection of objects near image edges.
- Height Estimation (Video Surveillance): The proposed method consistently yielded lower height estimation errors (in cm) compared to manual rectification methods, especially in scenarios with higher distortions and varying illumination. For example, average errors were as low as 0.39 cm on certain cameras, compared to manual methods with errors up to 1.69 cm.
Limitations and Future Work
- Computational Cost: While the method is automatic, the iterative refinement process can be computationally intensive, especially for very large images or complex scenes. The paper mentions processing time as a metric, and while improved over some methods, it's still a factor.
- Dependence on Line Features: The method relies on the presence of discernible line features in the image. Scenes lacking clear lines might pose a challenge.
- Future Work: The authors suggest further exploration of deep learning approaches for line segment extraction and potentially integrating their refinement scheme into end-to-end deep learning models for even greater efficiency and accuracy. They also highlight the need for continued validation on diverse real-world datasets.
Summary updated