Abstract
Existing deep learning-based interactive image segmentation approaches typically assume the target-of-interest is always a single object and fail to account for the potential diversity in user expectations, thus requiring excessive user input when it comes to segmenting an object part or a group of objects instead. Motivated by the observation that the object part, full object, and a collection of objects essentially differ in size, we propose a new concept called scale-diversity, which characterizes the spectrum of segmentations w.r.t. different scales. To address this, we present MultiSeg, a scale-diverse interactive image segmentation network that incorporates a set of two-dimensional scale priors into the model to generate a set of scale-varying proposals that conform to the user input. We explicitly encourage segmentation diversity during training by synthesizing diverse training samples for a given image. As a result, our method allows the user to quickly locate the closest segmentation target for further refinement if necessary. Despite its simplicity, experimental results demonstrate that our proposed model is capable of quickly producing diverse yet plausible segmentation outputs, reducing the user interaction required, especially in cases where many types of segmentations (object parts or groups) are expected.
The Challenge of Scale Ambiguity
Interactive image segmentation allows users to extract objects by providing "guide" clicks—positive for foreground and negative for background. However, most existing deep learning models suffer from a one-size-fits-all bias: they assume a click always refers to a single, complete object.
This creates a "scale ambiguity" problem. For instance, if a user clicks on a flower bouquet held by a bride, do they want just the flowers (a part), the bride (the object), or the entire wedding party (a group)? Standard models usually default to the "object" level. To select a small part or a larger group, a user must provide many tedious negative clicks to "deselect" the rest of the image. MultiSeg aims to provide diverse, plausible options immediately, allowing the user to pick the correct scale with minimal effort.
The MultiSeg Approach: Scale as a Proxy for Hierarchy
The authors introduce the concept of scale-diversity—characterizing the spectrum of possible segmentations based on their physical size and aspect ratio. The core insight is that semantic hierarchy (part → object → group) can be effectively modeled through spatial scale.
Architecture and Training
The model is built on a ResNet-101 based DeepLabv3+ backbone, modified to accept user clicks (transformed into distance maps) alongside the RGB image. Instead of a single output mask, the network features $M=12$ branches. Each branch is conditioned on a specific 2D scale prior—combinations of different sizes and aspect ratios. This ensures that the diversity is interpretable: each branch has a specific job, such as "find something of this specific size and shape around the click."
To teach the model that one click can mean many things, the authors developed a training pipeline that synthesizes hierarchical ground truths. For a given image, they group neighboring objects to create "group" targets and use instance annotations to define "part" targets.
Inference and Selection
In a single forward pass (<100ms), the model generates 12 masks. It then uses Non-Maximum Suppression (NMS) to filter out redundant proposals and presents the top $N$ (usually 3) candidates to the user. A small classifier predicts an "objectness score" for each scale to help rank these proposals.
Key Results and User Impact
MultiSeg was evaluated on standard benchmarks (GrabCut, Berkeley, PASCAL VOC) and specialized datasets like Fashionista to test part segmentation (e.g., segmenting a hat vs. a whole person).
- Reduced Interaction: On PASCAL VOC, MultiSeg achieved 85% IoU with fewer clicks than state-of-the-art models like DIOS and DEXTR.
- Superior Part Segmentation: On the Fashionista dataset, MultiSeg often provided specific clothing items as top proposals after the very first click, whereas standard models required multiple clicks to deselect the rest of the body.
- User Study: In tests with real humans, MultiSeg required the least amount of time (7.55s per task) and the fewest total interactions (3.54) compared to baselines (8.84s and 4.62 interactions for DIOS).
Conclusion
MultiSeg effectively solves the "What did you mean?" problem in interactive segmentation by embracing ambiguity rather than ignoring it. By leveraging 2D scale priors and diversity-aware training, it provides a fast, interpretable, and highly efficient tool for complex image editing tasks. Future work may involve incorporating depth information for even more semantically meaningful groupings and integrating boundary-aware refinement directly into the multi-scale architecture.
Summary updated