TL;DR
This paper introduces PA-SAM, a novel framework that adapts the powerful Segment Anything Model (SAM) for industrial defect segmentation. By combining a new parameter-efficient fine-tuning technique (MSPCA-LoRA) with an automated prompt generation system (IPEG), PA-SAM overcomes SAM's limitations in industrial settings, achieving state-of-the-art results on challenging defect detection tasks.
What Problem Does This Paper Address?
Industrial surface defect detection is crucial for product quality and manufacturing efficiency. Traditional methods often struggle with the complexity and variability of defects, leading to time-consuming manual inspections prone to subjective biases. While deep learning has shown promise, existing models often require extensive labeled data, which is costly and difficult to obtain for industrial scenarios. Vision foundation models like SAM offer impressive generalization capabilities but are not directly suited for industrial defect segmentation due to differences in data distribution (natural images vs. industrial defects), reliance on manual prompts, and a lack of category-specific information. This paper aims to bridge this gap by adapting SAM for robust and efficient industrial defect segmentation.
Methods
The proposed PA-SAM framework builds upon SAM's architecture with several key innovations:
- MSPCA-LoRA (Multi-Scale Partial Convolution Aggregation - Low-Rank Adaptation): This is a novel Parameter-Efficient Fine-Tuning (PEFT) technique. It integrates Multi-Scale Partial Convolution Aggregation (MSPCA) into the low-rank adaptation (LoRA) method. LoRA efficiently fine-tunes large models by decomposing weight matrices into smaller, trainable low-rank matrices. MSPCA enhances this by incorporating multi-scale convolutions and a partial channel fusion mechanism to better capture local features and address the limitations of ViT's global attention in focusing on fine-grained details. This approach aims to improve the image encoder's sensitivity to industrial defect characteristics while maintaining parameter efficiency.
- IPEG (Image-to-Prompt Embedding Generator): To eliminate the need for manual prompts (points, boxes), IPEG automatically generates high-quality prompt embeddings. It takes image embeddings as input and uses an Adaptive Multi-Scale Edge Enhancer (AMSEE), a Patch Decoder, and a Dense Encoder to create these embeddings, which then guide the mask decoder for segmentation. This automates a crucial step, making the system more practical for industrial applications.
- Multi-Class Mask Decoder: The original SAM decoder generates binary masks. This paper modifies the mask decoder to output segmentation maps corresponding to the number of defect categories present in the image, transforming SAM into an end-to-end multi-class semantic segmentation model suitable for industrial tasks where classification is important.
Datasets
The proposed PA-SAM framework was evaluated on two real-world industrial surface defect datasets:
- SD-Saliency-900 Dataset: This dataset contains challenging industrial images with complex scenarios, low contrast, and varying defect scales. It includes three defect types (patches, inclusions, scratches) with 300 images per type. The authors used 600 images for training and 300 for testing.
- Tire-Seg Dataset: This dataset is a standardized collection of X-ray images of tires for defect recognition. The original dataset had bounding box annotations, which were not directly suitable for SAM. The authors adapted it by performing pixel-level annotations for four typical defect types (cords-overlap, cords-distance, foreign-matter, cord-separate), using 860 images for training and 377 for testing. This dataset is noted for its complex surface structures and similar-looking defects, making it particularly challenging.
Key Insights and Ideas
- Bridging the Domain Gap: The core challenge addressed is the significant difference between the natural image data SAM was pre-trained on and the specific characteristics of industrial defect images. MSPCA-LoRA is designed to bridge this gap by making the model more sensitive to local, fine-grained features crucial for defect detection.
- Automating Prompts is Key: The reliance on manual prompts for SAM is a major bottleneck for industrial automation. The IPEG module is a critical innovation, demonstrating that high-quality prompt embeddings can be generated automatically from image features, significantly improving usability.
- PEFT for Industrial Adaptation: Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA are essential for adapting large foundation models to specialized tasks with limited data. The paper shows how combining PEFT with domain-specific feature extraction (MSPCA) can yield superior results.
- Beyond Binary Segmentation: For industrial applications, classifying defects is as important as segmenting them. Modifying SAM's mask decoder for multi-class output is a practical enhancement that unlocks its potential for more sophisticated defect analysis.
Key Results
PA-SAM demonstrated strong performance on both datasets, outperforming several state-of-the-art segmentation methods:
- SD-Saliency-900 Dataset: PA-SAM achieved a mean Intersection over Union (mIoU) of 73.87% and a mean Dice coefficient (mDice) of 84.90%. This was a significant improvement over other methods, with mIoU gains of up to 11.66% and mDice gains of up to 10.45%.
- Tire-Seg Dataset: PA-SAM achieved an mIoU of 68.30% and an mDice of 80.22%. Despite the dataset's challenges, PA-SAM still outperformed competing algorithms, showing performance gains of up to 7.05% in mIoU and 9.85% in mDice.
- Ablation Study: The study confirmed the effectiveness of individual modules. MSPCA-LoRA alone improved mIoU by over 22% and mDice by over 20% compared to a baseline SAM. The IPEG module also contributed positively, showing that even lightweight auxiliary modules can significantly boost performance.
Limitations and Future Work
- Limitations:
- While PA-SAM shows excellent generalization, the paper notes that addressing industrial defects with highly variable scales still presents an area for improvement.
- The trainable parameters for MSPCA-LORA and IPEG, while efficient, still represent a small fraction of the total model parameters, suggesting that further optimization might be possible.
- Future Work:
- The authors aim to further investigate SAM's application in other industrial vision domains, such as 3D vision and anomaly detection.
- Exploring even more lightweight adaptation strategies for SAM is suggested as a worthwhile future direction.
Summary updated