Abstract
X-Paste is a scalable data augmentation framework that addresses data scarcity in instance segmentation. By leveraging StableDiffusion for synthetic object generation and CLIP for filtering web-crawled images, X-Paste provides a virtually unlimited supply of high-quality, diverse object instances. This approach significantly improves performance on rare "long-tail" categories and enables robust open-vocabulary segmentation without requiring architectural changes to existing models.
Introduction
Instance segmentation requires extensive pixel-level annotations, which are costly and time-consuming to produce. This creates two primary challenges: the Long-Tail Problem, where rare categories lack sufficient training data, and Scalability, where traditional "Copy-Paste" augmentation is limited by the availability of human-annotated masks. X-Paste investigates whether generative AI and foundation models can automate the creation of an infinite training data pool.
Methodology
The X-Paste pipeline automates the acquisition and composition of object masks through four key modules.
- Object Instance Acquisition: Images are sourced via StableDiffusion (generating photorealistic images of specific categories) and web-crawling (filtered by CLIP to ensure category relevance).
- Instance Mask Generation: Class-agnostic models like U2Net or CLIPseg extract masks from these object-centric images.
- Instance Filtering: CLIP scores the similarity between the extracted mask and the category name to ensure quality.
- Instance Composition: Synthetic instances are pasted onto real backgrounds using a class-balanced sampling strategy.
Mask Quality and Filtering
The framework relies on the quality of automated segmentation. By comparing various models and using CLIP as a quality gate, the system ensures that only high-fidelity masks enter the training pool.
Experimental Results
The researchers evaluated X-Paste on the LVIS and MS-COCO datasets.
- LVIS: Using a CenterNet2 with a Swin-Large backbone, X-Paste achieved a +2.1 mask AP gain overall, with a significant +6.5 mask AP jump for rare classes.
- MS-COCO: The method reached 57.0 box AP and 48.6 mask AP.
- Open-Vocabulary: In zero-shot settings, X-Paste improved rare object detection by +5.0 mAP.
Discussion and Conclusion
X-Paste demonstrates that generative models can act as "equalizers" for long-tail distributions. The bottleneck in segmentation performance is often the variety and quality of training masks—the "oxygen" of the model—rather than the architecture itself. While dependencies on class-agnostic segmenters and the realism of blending remain areas for future work, X-Paste provides a powerful, plug-and-play strategy for scaling instance segmentation.
Summary updated