TL;DR
This paper introduces MLDAC, a novel self-supervised few-shot semantic segmentation method that leverages multi-task learning and dense attention computation. By cleverly splitting saliency masks and employing a Swin Transformer backbone with dense attention blocks, MLDAC significantly improves segmentation accuracy in scenarios with limited annotations, outperforming existing self-supervised methods.
What Problem Does This Paper Address?
Autonomous driving relies heavily on intelligent vehicles understanding their surroundings, with semantic segmentation playing a crucial role. Traditional methods require extensive pixel-level manual annotations, which are costly and time-consuming. While few-shot learning reduces this burden, it still often relies on supervised approaches. This paper tackles the challenge of achieving accurate semantic segmentation with minimal supervision, specifically in a few-shot setting, by developing a self-supervised framework that avoids the need for manual annotations altogether.
Methods
The core of MLDAC lies in its self-supervised, multi-task learning framework. The process begins with unsupervised saliency detection to identify salient regions within an image. This saliency mask is then split into two parts: one acts as a "support mask" for the few-shot segmentation task, while the other, along with the entire image, is used for a cross-entropy loss calculation. This multi-task approach helps improve the model's generalization ability.
The network architecture utilizes a Swin Transformer as a backbone for multi-scale feature extraction. These features are then fed into multiple levels of "Dense Attention Computation Blocks" (DACBs). These blocks are designed to enhance pixel-level correspondence by computing attention between query, key, and value features derived from different scales. Finally, an inter-scale mixing and up-sampling module aggregates these multi-scale attention results to produce the final segmentation mask. Data augmentation techniques like jittering, flipping, rotation, cropping, and Gaussian noise are also employed to enhance robustness and simulate real-world variations.
Datasets
The proposed method was evaluated on three standard datasets:
- PASCAL-5¹: A dataset derived from PASCAL VOC, split into four folds for few-shot evaluation.
- COCO-20¹: A subset of MS COCO, also divided into four folds for few-shot segmentation.
- FSS-1000: A dataset specifically designed for few-shot segmentation, featuring a large number of categories.
Key Insights and Ideas
- Self-supervised Saliency for Supervision: Using unsupervised saliency detection to generate "pseudo-labels" for both support and query masks is a clever way to bypass the need for manual annotations in a few-shot setting.
- Multi-task Learning for Generalization: Splitting the saliency mask and using different parts for distinct loss functions (one for direct segmentation guidance and another for broader region learning) effectively improves the model's ability to generalize to unseen classes.
- Dense Attention for Pixel-Level Correspondence: The DACB module's ability to compute attention across multiple feature scales is crucial for capturing fine-grained relationships between pixels, leading to more accurate segmentation boundaries.
- Swin Transformer as a Powerful Backbone: Leveraging a pre-trained Swin Transformer effectively extracts rich multi-scale semantic features, which are essential for the subsequent attention mechanisms.
Key Results
MLDAC demonstrated strong performance across the evaluated datasets:
- On PASCAL-5¹, it achieved 55.1% one-shot mIoU, outperforming many existing self-supervised methods and even surpassing some fully supervised approaches.
- On COCO-20¹, it achieved 26.8% one-shot mIoU.
- On FSS-1000, it reached 78.1% segmentation accuracy.
Qualitatively, visualizations showed that MLDAC could better delineate object boundaries and distinguish foreground from background, especially in complex scenes, compared to methods like MaskSplit.
Limitations and Future Work
The authors acknowledge that while their self-supervised method is effective, it still cannot perfectly capture continuous salient regions for objects within the same category. For future work, they propose introducing an image generation scheme to construct a more robust meta-learning paradigm for few-shot semantic segmentation, aiming for even higher segmentation accuracies.
Summary updated