Abstract
Standard semantic segmentation models typically treat images as independent and identically distributed (i.i.d.), ignoring the rich inter-image correlations found in personal photo galleries. This paper introduces Personalized Image Semantic Segmentation (PSS), a novel task and dataset comprising 10,080 images from 15 users. We propose a framework that leverages these correlations through a Group Context Module (GCM), which aggregates regional representations across related images in a user's gallery. Our approach significantly improves segmentation accuracy on unlabeled personal data, outperforming existing unsupervised domain adaptation methods.
Introduction
In the real world, particularly in mobile photography, images are rarely isolated. A single user’s gallery often contains recurring themes, objects, and environments—such as the same pet, car, or living room—captured across multiple shots. Traditional models, trained on diverse public datasets like PASCAL VOC, struggle with the distribution gap between polished training data and personal photos. Furthermore, they fail to utilize the "personalized traits" that could resolve visual ambiguities in difficult shots. This research addresses these gaps by treating a user's gallery as a structured context rather than a collection of independent samples.
Methodology
The proposed PSS framework combines Unsupervised Domain Adaptation (UDA) with a specialized context-aware architecture.
- Adversarial Domain Adaptation: To bridge the gap between labeled source data (PASCAL VOC) and unlabeled target personal data, the model employs an adversarial training strategy.
- Group Context Module (GCM): This is the core innovation. The model clusters a user’s images into semantic groups (e.g., "indoor pets"). It then extracts "soft" regional representations for object classes within these groups. When segmenting a specific pixel, an attention mechanism allows the model to "consult" similar regions across other images in the same group, effectively borrowing confidence from clearer frames to resolve blurry or obscured shapes.
- Pseudo-Label Refinement: The model identifies high-confidence "easy" images based on low entropy and uses their predictions as pseudo-labels to further fine-tune the network.
The PSS Dataset
The authors curated the PSS dataset to facilitate this new task. It contains 10,080 images from 15 volunteers, covering the same 20 classes as PASCAL VOC. Unlike standard datasets, PSS reflects real-world "long-tailed" distributions; for instance, a user may have hundreds of photos of their dog but none of an aeroplane. Approximately 30% of the dataset is manually annotated with pixel-level masks for validation purposes.
Experimental Results
The proposed method, OURS-S2, consistently outperformed state-of-the-art domain adaptation baselines such as ADVENT and MRNet. Using a ResNet-50 backbone, the model achieved a Mean Foreground IoU (FIoU) of 59.72, compared to 56.59 for ADVENT.
Qualitative results demonstrate that the Group Context Module produces significantly cleaner masks. By leveraging inter-image context, the model can accurately segment objects that are partially obscured or in poor lighting by referencing clearer instances of the same object within the user's group.
Conclusion and Future Work
Personalization in semantic segmentation is most effective when handled via local inter-image context rather than global style extraction. While the current K-means clustering approach and performance on extremely rare classes leave room for improvement, this work demonstrates that the inherent correlations in personal galleries are a powerful, untapped resource for computer vision. Future research will explore more flexible clustering strategies to better accommodate diverse user behaviors.
Summary updated