Where is My Mug? Training-Free Framework for Personalized Image Segmentation

Quang Qui-Vinh Nguyen, Hung Nguyen, Rang Nguyen, Khoi NguyenarXiv 2026 · arXiv

TL;DR

This paper introduces a training-free, modular framework for Personalized Image Segmentation (PIS) that successfully distinguishes a specific object instance from visually similar "look-alikes." By shifting to a "recall-first" architecture and introducing a robust, high-speed matching algorithm called top-$K$ Sinkhorn, the authors achieve a massive performance leap—up to 48 mIoU—over existing methods on a new, more realistic multi-instance benchmark.

What Problem Does This Paper Address?

Imagine asking a robot, "Where is my mug?" in a kitchen filled with several nearly identical green ceramic mugs. Standard semantic segmentation would find all "mugs," and few-shot segmentation (FSS) would likely do the same. PIS, however, requires identifying the exact physical instance shown in a reference image, despite changes in lighting, viewpoint, or partial occlusion.

The authors argue that current PIS research is hampered by "easy" benchmarks that typically feature only one instance of a category per image. In these simplified settings, existing methods like PerSAM look great. However, when faced with realistic clutter and multiple similar objects, these methods "collapse" because they fail to even propose the correct object as a candidate. The core issue isn't the quality of the final mask, but a recall bottleneck in the initial localization phase.

Figure 1. The difference between class-level (FSS) and identity-level (PIS) segmentation.

Methods

The authors propose a modular, three-stage pipeline that requires no task-specific training:

  1. Class-Agnostic Object Proposals: Instead of relying on heatmaps or point-prompts, the system uses an off-the-shelf detector (like YOLO-World) to generate a high-recall set of bounding box candidates for all potential objects in the scene.
  2. Reference-Candidate Matching: This is the heart of the paper. They extract features using foundation models (like BLIP-2 or DINOv2). To compare a candidate box to the reference, they use a novel top-$K$ Sinkhorn matching algorithm.
  3. Standard patch-level matching (like Sinkhorn divergence) gets "diluted" by background noise and shared category features.
  4. Top-$K$ Sinkhorn solves this by only scoring a candidate based on its most confidently matched patches (the "inliers"). It acts as a computationally efficient surrogate for partial optimal transport, running over 100$\times$ faster while maintaining the same robustness.
  5. Mask Refinement: The best-matching bounding box is fed into the Segment Anything Model (SAM) to produce the final high-quality segmentation mask.
Figure 3. The proposed modular PIS pipeline.

Datasets

The study evaluates performance across several benchmarks:

  • PerSeg, PerMIS, and DAVIS: Existing datasets often used for PIS, though noted for being largely single-instance or having minimal variation.
  • PerMultiSeg (New): A challenging new benchmark introduced in this paper. It contains 1,079 target images and 2,147 reference images across 45 categories. Crucially, it averages nearly 4 same-category, same-color objects per image, forcing the model to rely on fine-grained visual cues (like specific scuffs or labels) rather than just category semantics.
Figure 2. Comparison of PerMultiSeg against existing, simpler PIS benchmarks.

Key Insights and Ideas

  • Recall is King: The "aha moment" was realizing that prior failures weren't due to bad masks, but because the correct object was never "surfaced" by point-prompting methods. A "recall-first" design using bounding boxes is much more reliable.
  • The "Look-Alike" Dilution: In multi-instance scenes, standard matching metrics fail because the common features between two different mugs (the "mug-ness") overwhelm the subtle differences that define their unique identities.
  • Efficiency through Trimming: By simply sorting and trimming the dual potentials of a standard Sinkhorn solve, the authors recovered the robustness of complex partial optimal transport at a fraction of the cost, making it practical for real-time deployment.
  • Foundation Models are Sufficient: The results show that we don't need to train new PIS models; we just need better ways to compose existing foundation models (detectors, encoders, and SAM).

Key Results

  • Massive Accuracy Gains: On the new PerMultiSeg benchmark, the proposed system achieved 80.07 mIoU, outperforming the previous state-of-the-art, $\mathrm{P}^2\mathrm{SAM}$, by over 48 points.
  • Competitive Speed: Despite the more robust matching, the system maintains an end-to-end inference time of approximately 1.02 seconds per image, matching the runtime of much less accurate baselines.
  • Ablation Success: The top-$K$ Sinkhorn matcher alone provided a +2.93 mIoU boost over full Sinkhorn matching, proving the value of the inlier-restricted approach.
Figure 1. An example from our challenging PerMultiSeg dataset. Given a distinctive green ceramic mug in the reference image, the task is to accurately identify and segment the same mug among similar ones in the target image. While existing PIS methods such as PerSAM [57] fail, our method successfully distinguishes and segments the correct mug. In contrast, few-shot segmentation (FSS) methods would segment all mugs in the scene, without differentiating the specific instance.

Limitations and Future Work

  • Terminal Detector Misses: The pipeline has no recovery mechanism; if the initial class-agnostic detector misses the object, the system cannot find it.
  • Discriminative Occlusion: If the specific feature that distinguishes an object (e.g., a small logo) is occluded while a distractor shows a similar feature, the matcher will likely fail.
  • "In-the-wild" References: The current evaluation uses clean, background-removed reference crops. Future work could involve more challenging "in-the-wild" references like loose hand-drawn boxes or messy screenshots.
  • Rejection Option: Adding a "no-match" rejection capability would be a natural next step for real-world robotic applications where the target object might not be present in the scene at all.

Summary updated

Abstract

\textbf{Personalized Image Segmentation (PIS)} aims to segment the specific object in a target image that matches a user-provided reference. Unlike category-based segmentation or few-shot segmentation, PIS must discriminate between multiple visually similar instances, a requirement central to deployed applications such as augmented reality, personalized content editing, robotics manipulation, inventory tracking, and fine-grained visual retrieval. We show that the strong numbers reported by existing PIS methods are largely an artifact of single-instance benchmarks: when evaluated on realistic scenes with several same-category look-alikes, prior point-prompt methods collapse, because the correct instance is frequently never surfaced as a prompt. We trace this to a recall bottleneck rather than a mask-quality one, and address it with a training-free, modular system. The pipeline first generates high-recall object proposals using an off-the-shelf class-agnostic detector, then matches each proposal to the reference using foundation-model features (e.g., BLIP-2), and finally refines the selected box with SAM. For robust matching we propose \textbf{top-$K$ Sinkhorn}, a patch-level distance that scores each candidate only on its most confidently matched patches; trimming the Sinkhorn dual potentials recovers the robustness of partial optimal transport at more than $100\times$ lower cost, keeping it inside the per-proposal candidate loop. To evaluate this regime, we introduce \textbf{\datasetname}, a realistic multi-instance benchmark. Across \datasetname, PerSeg, PerMIS, and DAVIS, our system surpasses PerSAM, $\mathrm{P}^2\mathrm{SAM}$, and PDM by up to $\sim\!48$ mIoU while matching their runtime and using fewer parameters, establishing a strong, deployable training-free baseline for real-world personalized segmentation.
BibTeX
@misc{nguyen2026where,
  author = {Quang Qui-Vinh Nguyen and Hung Nguyen and Rang Nguyen and Khoi Nguyen},
  title = {{Where is My Mug? Training-Free Framework for Personalized Image Segmentation}},
  year = {2026},
  eprint = {2611.15216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15216},
}
Figure