Personalized Image Semantic Segmentation

Yu Zhang, Chang-Bin Zhang, Peng-Tao Jiang, Feng Mao, Ming-Ming Cheng

Abstract

Standard semantic segmentation models typically treat images as independent and identically distributed (i.i.d.), ignoring the rich inter-image correlations found in personal photo galleries. This paper introduces Personalized Image Semantic Segmentation (PSS), a novel task and dataset comprising 10,080 images from 15 users. We propose a framework that leverages these correlations through a Group Context Module (GCM), which aggregates regional representations across related images in a user's gallery. Our approach significantly improves segmentation accuracy on unlabeled personal data, outperforming existing unsupervised domain adaptation methods.

Introduction

In the real world, particularly in mobile photography, images are rarely isolated. A single user’s gallery often contains recurring themes, objects, and environments—such as the same pet, car, or living room—captured across multiple shots. Traditional models, trained on diverse public datasets like PASCAL VOC, struggle with the distribution gap between polished training data and personal photos. Furthermore, they fail to utilize the "personalized traits" that could resolve visual ambiguities in difficult shots. This research addresses these gaps by treating a user's gallery as a structured context rather than a collection of independent samples.

Methodology

The proposed PSS framework combines Unsupervised Domain Adaptation (UDA) with a specialized context-aware architecture.

  1. Adversarial Domain Adaptation: To bridge the gap between labeled source data (PASCAL VOC) and unlabeled target personal data, the model employs an adversarial training strategy.
  2. Group Context Module (GCM): This is the core innovation. The model clusters a user’s images into semantic groups (e.g., "indoor pets"). It then extracts "soft" regional representations for object classes within these groups. When segmenting a specific pixel, an attention mechanism allows the model to "consult" similar regions across other images in the same group, effectively borrowing confidence from clearer frames to resolve blurry or obscured shapes.
  3. Pseudo-Label Refinement: The model identifies high-confidence "easy" images based on low entropy and uses their predictions as pseudo-labels to further fine-tune the network.
The proposed PSS framework featuring the Group Context Module and pseudo-label refinement.

The PSS Dataset

The authors curated the PSS dataset to facilitate this new task. It contains 10,080 images from 15 volunteers, covering the same 20 classes as PASCAL VOC. Unlike standard datasets, PSS reflects real-world "long-tailed" distributions; for instance, a user may have hundreds of photos of their dog but none of an aeroplane. Approximately 30% of the dataset is manually annotated with pixel-level masks for validation purposes.

Experimental Results

The proposed method, OURS-S2, consistently outperformed state-of-the-art domain adaptation baselines such as ADVENT and MRNet. Using a ResNet-50 backbone, the model achieved a Mean Foreground IoU (FIoU) of 59.72, compared to 56.59 for ADVENT.

Qualitative results demonstrate that the Group Context Module produces significantly cleaner masks. By leveraging inter-image context, the model can accurately segment objects that are partially obscured or in poor lighting by referencing clearer instances of the same object within the user's group.

Qualitative comparison showing the superiority of the proposed method in handling complex personal scenes compared to standard domain adaptation baselines.

Conclusion and Future Work

Personalization in semantic segmentation is most effective when handled via local inter-image context rather than global style extraction. While the current K-means clustering approach and performance on extremely rare classes leave room for improvement, this work demonstrates that the inherent correlations in personal galleries are a powerful, untapped resource for computer vision. Future research will explore more flexible clustering strategies to better accommodate diverse user behaviors.

Summary updated

Abstract

Semantic segmentation models trained on public datasets have achieved great success in recent years. However, these models didn't consider the personalization issue of segmentation though it is important in practice. In this paper, we address the problem of personalized image segmentation. The objective is to generate more accurate segmentation results on unlabeled personalized images by investigating the data's personalized traits. To open up future research in this area, we collect a large dataset containing various users' personalized images called PIS (Personalized Image Semantic Segmentation). We also survey some recent researches related to this problem and report their performance on our dataset. Furthermore, by observing the correlation among a user's personalized images, we propose a baseline method that incorporates the inter-image context when segmenting certain images. Extensive experiments show that our method outperforms the existing methods on the proposed dataset. The code and the PIS dataset will be made publicly available.
BibTeX
@inproceedings{zhang2021personalized,
  author = {Yu Zhang and Chang-Bin Zhang and Peng-Tao Jiang and Feng Mao and Ming-Ming Cheng},
  title = {{Personalized Image Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01038},
}
Figure