X-Paste: Revisit Copy-Paste at Scale with CLIP and StableDiffusion

Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen 0001, Dong Chen 0003, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, Weiming Zhang, Nenghai Yu

Abstract

X-Paste is a scalable data augmentation framework that addresses data scarcity in instance segmentation. By leveraging StableDiffusion for synthetic object generation and CLIP for filtering web-crawled images, X-Paste provides a virtually unlimited supply of high-quality, diverse object instances. This approach significantly improves performance on rare "long-tail" categories and enables robust open-vocabulary segmentation without requiring architectural changes to existing models.

Introduction

Instance segmentation requires extensive pixel-level annotations, which are costly and time-consuming to produce. This creates two primary challenges: the Long-Tail Problem, where rare categories lack sufficient training data, and Scalability, where traditional "Copy-Paste" augmentation is limited by the availability of human-annotated masks. X-Paste investigates whether generative AI and foundation models can automate the creation of an infinite training data pool.

Methodology

The X-Paste pipeline automates the acquisition and composition of object masks through four key modules.

The X-Paste pipeline: (a) Object Instance Acquisition via CLIP filtering and StableDiffusion, (b) Instance Mask Generation, (c) Instance Filtering, and (d) Instance Composition.
  1. Object Instance Acquisition: Images are sourced via StableDiffusion (generating photorealistic images of specific categories) and web-crawling (filtered by CLIP to ensure category relevance).
  2. Instance Mask Generation: Class-agnostic models like U2Net or CLIPseg extract masks from these object-centric images.
  3. Instance Filtering: CLIP scores the similarity between the extracted mask and the category name to ensure quality.
  4. Instance Composition: Synthetic instances are pasted onto real backgrounds using a class-balanced sampling strategy.

Mask Quality and Filtering

The framework relies on the quality of automated segmentation. By comparing various models and using CLIP as a quality gate, the system ensures that only high-fidelity masks enter the training pool.

Comparison of different class-agnostic segmentation models (SRF, CLIPseg, UFO, U2Net) and their corresponding CLIP scores used for quality filtering.

Experimental Results

The researchers evaluated X-Paste on the LVIS and MS-COCO datasets.

  • LVIS: Using a CenterNet2 with a Swin-Large backbone, X-Paste achieved a +2.1 mask AP gain overall, with a significant +6.5 mask AP jump for rare classes.
  • MS-COCO: The method reached 57.0 box AP and 48.6 mask AP.
  • Open-Vocabulary: In zero-shot settings, X-Paste improved rare object detection by +5.0 mAP.

Discussion and Conclusion

X-Paste demonstrates that generative models can act as "equalizers" for long-tail distributions. The bottleneck in segmentation performance is often the variety and quality of training masks—the "oxygen" of the model—rather than the architecture itself. While dependencies on class-agnostic segmenters and the realism of blending remain areas for future work, X-Paste provides a powerful, plug-and-play strategy for scaling instance segmentation.

Summary updated

BibTeX
@inproceedings{zhao2023x,
  author = {Hanqing Zhao and Dianmo Sheng and Jianmin Bao and Dongdong Chen and Dong Chen and Fang Wen and Lu Yuan and Ce Liu and Wenbo Zhou and Qi Chu and Weiming Zhang and Nenghai Yu},
  title = {{X-Paste: Revisit Copy-Paste at Scale with CLIP and StableDiffusion}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=6h2Ydbr8SI},
}
Figure