A Generative Approach at the Instance-Level for Image Segmentation Under Limited Training Data Conditions (Student Abstract)

Thanh-Danh Nguyen, Vinh-Tiep Nguyen, Tam V. NguyenAAAI 2025 · DOI

Abstract

Instance segmentation models typically require vast amounts of pixel-level annotated data to achieve high performance. In data-scarce environments, these models often fail to generalize. This paper proposes a generative approach to mitigate "data hunger" by synthesizing new training samples at the instance level and embedding generative feature extractors directly into segmentation pipelines. Experimental results on the Cityscapes benchmark demonstrate a significant performance boost, with an average increase of 12.04% in Average Precision (AP) under limited data conditions.

Introduction

High-performance instance segmentation—the task of identifying and outlining every individual object in an image—is traditionally a data-intensive endeavor. Manually labeling images at the pixel level is both slow and expensive. When researchers operate in "data-scarce" environments with only a few hundred or thousand images, standard deep learning models struggle. While Generative AI (GenAI) has shown promise in creating synthetic data for semantic segmentation, applying these techniques at the instance level remains challenging. This work explores "distilling" the visual knowledge within pre-trained diffusion models to enhance discriminative models when real-world data is limited.

Methodology

The proposed framework employs a two-pronged strategy to empower instance segmentation:

  1. Instance-level Synthesis (Data Augmentation): Instead of generating entire scenes, specific object instances and their masks are extracted from the limited training set. These are fed into an Instance-wise Image Generator ($\Theta_G$), specifically the BlendedDiff architecture. By combining the original mask with a class-guided text prompt, the generator creates diverse, high-quality variations of the object. These synthetic "twins" effectively multiply the training set's size and diversity.
  2. Embedded Generative Architectures: To avoid the computational overhead of full image rendering, the authors propose embedding the feature extractor $e(\Theta_G)$ from a pre-trained generative model directly into the segmentation model. This allows the segmentation pipeline to leverage the generative model's internal understanding of object boundaries and textures.
Overview of the proposed generative instance segmentation framework, showing (a) instance-wise synthesis and (b) generative extractor embedding.

Experimental Results

The approach was evaluated using the Cityscapes benchmark, utilizing approximately 2.5K annotated images. The quality of generated images was measured using CLIPScore and FID, while segmentation accuracy was assessed via Mean Average Precision (mAP).

Among the tested diffusion models (GLIGEN, DiffInpainting, and BlendedDiff), BlendedDiff achieved the highest quality with a CLIPScore of 0.87 and an FID of 16.28. More importantly, integrating the generative approach led to substantial gains in segmentation performance across different backbones.

Performance comparison of FastInst and OneFormer models with and without generative synthesis on the Cityscapes dataset.

As shown in the results, the FastInst model (R50-FPN backbone) improved from 27.65 AP to 36.52 AP. The OneFormer model (ConvNext-L backbone) saw an even more dramatic increase, rising from 21.75 AP to 38.93 AP. On average, the generative augmentation provided a 12.04% AP improvement.

Conclusion and Future Work

This research demonstrates that treating generative models like Stable Diffusion as massive repositories of visual knowledge can significantly benefit discriminative tasks like instance segmentation. By focusing synthesis at the instance level, the model avoids background noise and focuses on relevant object features.

Future work will focus on optimizing these large generative components for real-time use and investigating the effectiveness of this approach in extreme "few-shot" settings, such as camouflaged object detection where data is exceptionally scarce.

Summary updated

BibTeX
@inproceedings{nguyen2025generative,
  author = {Thanh-Danh Nguyen and Vinh-Tiep Nguyen and Tam V. Nguyen},
  title = {{A Generative Approach at the Instance-Level for Image Segmentation Under Limited Training Data Conditions (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35284},
}
Figure