TL;DR
This paper introduces FishSegSSL, a novel semi-supervised learning framework for semantic segmentation of fish-eye images, a challenging task due to data scarcity and image distortion. By adapting and enhancing existing semi-supervised techniques with pseudo-label filtering, dynamic confidence thresholding, and robust strong augmentation, FishSegSSL achieves significant performance improvements over fully supervised methods on the WoodScape dataset.
What Problem Does This Paper Address?
Computer vision systems, particularly those used in autonomous driving, increasingly rely on wide field-of-view (FoV) cameras equipped with fish-eye lenses to capture a broad scene. While deep learning has revolutionized semantic segmentation for standard perspective images, its application to fish-eye imagery is hampered by a lack of large, annotated datasets. Existing methods often resort to transforming fish-eye images into perspective views, which can lead to information loss at the edges, or rely on synthetic data that doesn't generalize well to real-world scenarios. Furthermore, multi-modal or multi-task approaches, while effective, are computationally expensive and require even more complex data collection. This paper addresses the critical need for effective semantic segmentation directly on real-world fish-eye images, specifically by leveraging the abundant unlabeled data through semi-supervised learning (SSL).
Methods
The authors propose FishSegSSL, a novel framework built upon the Cross-Pseudo-Supervision (CPS) method, enhanced with three key semi-supervised learning components:
- Pseudo-Label Filtering: This component addresses the issue of noisy pseudo-labels generated by SSL methods. It filters out low-confidence predictions, thereby reducing the noise injected into the learning signal from unlabeled data.
- Dynamic Confidence Thresholding: Moving beyond fixed thresholds, this technique dynamically adjusts the confidence threshold for pseudo-label filtering based on the learning difficulty of individual classes. This adaptive approach ensures more consistent learning across different object categories.
- Robust Strong Augmentation (RSA): This module aims to further utilize "easy" samples that are confidently classified. It applies a hard augmentation (a combination of various augmentations like brightness, color adjustment, and jitter) to these samples, acting as a regularizer to prevent overfitting and encourage continued learning.
These components are integrated into a uni-modal, single-task framework designed to work directly with fish-eye images.
Datasets
The primary dataset used for training and evaluation is WoodScape. This dataset is specifically designed for autonomous driving and features multi-camera fish-eye images with annotations for nine essential tasks, including semantic segmentation. The authors note that WoodScape is one of the first labeled datasets specifically designed for semantic segmentation in this domain. For their experiments, they utilized 20% of the 8029 samples as the labeled set and the remainder as the unlabeled set for semi-supervised training.
Key Insights and Ideas
- Fish-eye distortion is a significant challenge for standard SSL: Existing SSL methods, originally designed for perspective images, perform suboptimally on fish-eye data due to the inherent radial distortion.
- Dynamic thresholding is crucial for class imbalance: Different classes have varying learning difficulties. A dynamic threshold that adapts to class-specific learning status is more effective than a fixed threshold for filtering pseudo-labels.
- Robust augmentation for easy samples matters: While seemingly already learned, "easy" samples can still benefit from strong augmentation to prevent overfitting and improve generalization.
- Uni-modal, single-task SSL is viable and effective: The proposed FishSegSSL demonstrates that it's possible to achieve high performance without relying on multi-modal data or complex multi-task learning setups for fish-eye images.
Key Results
FishSegSSL achieved a mean intersection over union (mIoU) of 64.81% on the WoodScape dataset. This represents:
- A 10.49% improvement over the fully supervised baseline using the same amount of labeled data.
- A 2.34% improvement over the best-performing existing semi-supervised method (CPS with CutMix augmentation).
The ablation study highlighted the individual contributions of the proposed components, with dynamic confidence thresholding showing a particularly significant impact.
Limitations and Future Work
The authors acknowledge that their work focused on a subset of existing SSL literature designed for perspective images. They suggest that exploring other SSL methods already established in the field could further enhance performance on fish-eye images. They also express hope that their work will encourage the development of more specialized techniques for semantic segmentation and other computer vision tasks related to fish-eye images, especially in data-scarce scenarios.
Summary updated