Structured Click Control in Transformer-based Interactive Segmentation

Long Xu, Yongquan Chen, Rui Huang, Feng Wu, Shiwu LaiarXiv 2024 · DOI · arXiv

Abstract

Interactive segmentation relies on user-provided clicks to refine object masks. While Vision Transformer (ViT) backbones have advanced the state-of-the-art, they often struggle with "control"—where additional clicks fail to improve or even degrade the segmentation result. We present Structured Click Control (SCC), a plug-and-play module that models user intent through Graph Neural Networks (GNNs). By treating clicked image tokens as graph nodes and integrating their features via dual cross-attention, SCC enhances the robustness and predictability of interactive models. Our evaluation across standard and cross-domain datasets demonstrates significant reductions in the number of clicks required to achieve high-quality masks.

Introduction

Interactive segmentation is a critical tool for efficient image annotation and object extraction. Users guide the model using positive (foreground) and negative (background) clicks. Modern architectures, such as the Segment Anything Model (SAM) and FocalClick, leverage Vision Transformers to achieve impressive zero-shot performance. However, these models frequently encounter a "stalling" problem: as the number of clicks increases, the segmentation mask may become unresponsive to new input or exhibit erratic behavior.

This lack of control stems from how clicks are typically processed. Most current methods represent clicks as simple two-channel spatial heatmaps concatenated with the input image. This approach fails to capture the semantic and structural relationships between the user's intent and the global image features. In complex scenarios, such as remote sensing or thin-object segmentation, the model's reliance on local spatial cues leads to suboptimal convergence.

Methodology: Structured Click Control (SCC)

To address the control problem, we propose the Structured Click Intent Model. Instead of treating clicks as mere spatial coordinates, SCC models the interaction within the feature space of the Transformer.

The overall architecture of the SCC module, showing its integration into single-scale and multi-scale Vision Transformers.

The SCC module consists of three primary components:

  1. Graph Node Selection: The system identifies the most relevant image tokens to represent the user's intent. It calculates the similarity between the features of user-clicked tokens and all other tokens in the image. A "top-k selection" strategy is then employed to designate these high-similarity tokens as nodes in a graph.
  2. GNN Fusion: The selected nodes are processed using a Graph Neural Network. The authors explored both Graph Convolutional Networks (GCN) and Graph Attention Networks (GAT). Findings indicate that GAT is superior because it dynamically learns the importance of edges, allowing the model to adapt to sparse and varied user inputs more effectively than the fixed adjacency matrices used in GCNs.
  3. Dual Cross-Attention: The refined "intent" features from the GNN are injected back into the primary vision Transformer features using a dual cross-attention mechanism. This ensures that the segmentation head receives explicit guidance from the structured user cues throughout the decoding process.

SCC is designed to be a generic enhancement, compatible with both single-scale backbones (e.g., SAM, SimpleClick) and multi-scale backbones (e.g., FocalClick).

Experiments and Results

The models were trained on the massive COCO + LVIS dataset and evaluated across a wide variety of standard benchmarks, including general datasets (GrabCut, Berkeley, SBD, DAVIS) and cross-domain remote sensing datasets (LoveDA, Rice).

Quantitative Performance

The primary metric for interactive segmentation is the Number of Clicks (NoC) required to reach a target Intersection over Union (IoU), typically 90%. SCC consistently reduced the NoC across all tested backbones and datasets. For instance, on the SBD dataset, adding SCC (GAT) to SimpleClick-ViT-B reduced the NoC 90 from 5.62 to 4.88. On the DAVIS dataset, FocalClick-B3-S2 + GAT showed a dramatic improvement, reducing the NoC 90 from 5.59 to 3.77.

Quantitative comparison of mIoU scores versus the number of clicks across various datasets and backbones. SCC (GAT) consistently outperforms baseline models.

Qualitative Robustness

SCC significantly improves the model's responsiveness in challenging scenarios. While baseline models often get "stuck" on background noise or fail to capture thin structures even with multiple clicks, the SCC-enhanced versions converge faster and more accurately. This is particularly evident in cross-domain tasks like remote sensing, where the model must generalize to unfamiliar data distributions.

Conclusion

Structured Click Control (SCC) provides a robust solution to the control problem in interactive segmentation. By leveraging GNNs to model user intent in the feature space, SCC ensures that every click contributes meaningfully to the final mask. The module is lightweight, adding only ~1.3% more parameters to large backbones like HQ-SAM, while providing substantial gains in efficiency and predictability.

Summary updated

BibTeX
@misc{xu2024structured,
  author = {Long Xu and Yongquan Chen and Rui Huang and Feng Wu and Shiwu Lai},
  title = {{Structured Click Control in Transformer-based Interactive Segmentation}},
  year = {2024},
  eprint = {2405.04009},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.04009},
}
Figure