SPT: Sequence Prompt Transformer for Interactive Image Segmentation

Senlin Cheng, Haopeng Sun, Tao Xie, Hangyue Zhao, Yiqiang Chen, Bolei Xu, Xiaobo LiICASSP 2025 · OpenReview

TL;DR

This paper introduces the Sequence Prompt Transformer (SPT), a novel method for interactive image segmentation that leverages information from a sequence of related images to improve accuracy. By incorporating a Top-k Prompt Selection (TPS) module and a new ADE20K-Seq dataset, SPT significantly outperforms existing single-image segmentation approaches, especially in scenarios requiring consistent object segmentation across multiple frames.

What Problem Does This Paper Address?

Interactive image segmentation is a crucial task where users provide input (like clicks) to guide an algorithm in precisely segmenting an object of interest. While effective for single images, real-world applications often involve segmenting the same object across a series of images (e.g., tracking a car window through a video sequence). Existing methods treat each image independently, failing to utilize the rich contextual information available in sequential frames. This leads to inefficiencies and potential errors, as corrections must be made for each image individually. The paper addresses this gap by proposing a method that explicitly leverages sequential image data, clicks, and masks to achieve more accurate and efficient segmentation over time.

Methods

The core of the proposed method is the Sequence Prompt Transformer (SPT). It operates on a sequence of images, where each image is accompanied by user-provided clicks and the predicted mask from the previous step.

  1. Backbone: A Vision Transformer (ViT) is used as the backbone to extract features from individual images, clicks, and masks.
  2. Top-k Prompt Selection (TPS): To avoid overwhelming the model with too much sequential data, a TPS module is introduced. It uses DINOv2 features to identify the k most similar images, clicks, and masks from the sequence to serve as effective prompts for the current image. This ensures that only the most relevant contextual information is used.
  3. Sequence Prompt Transformer (SPT): This module processes the sequence of features. A key innovation here is the Multi-head Concealed Self-Attention mechanism. This ensures that the model can only attend to information from previous frames and the current frame, preventing it from "cheating" by looking at future information in the sequence. This is achieved by using a masking strategy based on position coding.
  4. Feature Pyramid Module (FPM) and MLP Segmentation Head (MSH): These components process the output of the SPT to generate the final segmentation mask, integrating multi-scale features for fine details.

Datasets

  • Training: COCO [35] and LVIS [36] datasets were used for training.
  • Evaluation: The model was evaluated on existing benchmarks: GrabCut [32], Berkeley [33], COCO-MVal [35], and DAVIS [34].
  • New Benchmark: The paper introduces ADE20K-Seq, a new dataset specifically designed for sequential interactive image segmentation. It is derived from ADE20K and comprises seven category-specific benchmarks, each with over 100 images, to better evaluate methods that leverage sequential information.

Key Insights and Ideas

  • Sequential Context is Key: The most significant insight is that leveraging information from previous frames in a sequence dramatically improves interactive segmentation accuracy and efficiency, especially for objects that remain consistent across frames.
  • Prompt Engineering for Sequences: Simply feeding all previous frames is impractical. The Top-k Prompt Selection (TPS) module is crucial for intelligently selecting the most informative prior frames, clicks, and masks, making the sequential approach efficient and effective.
  • "Concealed" Attention for Temporal Integrity: The "Concealed Self-Attention" mechanism is a clever way to enforce causality in temporal modeling, ensuring that the model learns from the past without peeking into the future, which is essential for realistic sequential processing.
  • Bridging the Dataset Gap: The creation of ADE20K-Seq addresses a critical need for benchmarks that can properly evaluate sequential interactive segmentation methods, moving beyond single-image evaluations.

Key Results

  • The proposed SPT method consistently outperforms state-of-the-art single-image interactive segmentation methods across multiple benchmark datasets, including the newly introduced ADE20K-Seq.
  • The paper reports significant improvements in segmentation accuracy, particularly when using fewer clicks, highlighting the efficiency gains from sequential context.
  • Ablation studies confirm the effectiveness of both the SPT module and the TPS prompt selection strategy.

Limitations and Future Work

  • Prompt Sequence Length: While experiments show performance improves with longer prompt sequences, the paper doesn't explicitly define an optimal length, suggesting further exploration.
  • Computational Cost: While the MSH uses lightweight MLP layers, processing sequences inherently adds computational overhead compared to single-image methods. The paper doesn't detail the exact speedup or slowdown.
  • Generalizability to Highly Dynamic Sequences: The current approach is most effective when objects remain relatively consistent across frames. Future work could explore adapting SPT to more dynamic scenarios where objects change significantly or appear/disappear frequently.
  • User Interaction Simulation: The paper mentions using a click simulation strategy from RITM. Further research could explore more sophisticated or realistic user interaction models.

Summary updated

Abstract

Interactive segmentation aims to extract objects of interest from an image based on user-provided clicks. In real-world applications, there is often a need to segment a series of images featuring the same target object. However, existing methods typically process one image at a time, failing to consider the sequential nature of the images. To overcome this limitation, we propose a novel method called Sequence Prompt Transformer (SPT), the first to utilize sequential image information for interactive segmentation. Our model comprises two key components: (1) Sequence Prompt Transformer (SPT) for acquiring information from sequence of images, clicks and masks to improve accurate. (2) Topk Prompt Selection (TPS) selects precise prompts for SPT to further enhance the segmentation effect. Additionally, we create the ADE20K-Seq benchmark to better evaluate model performance. We evaluate our approach on multiple benchmark datasets and show that our model surpasses state-of-the-art methods across all datasets.
BibTeX
@inproceedings{cheng2025spt,
  author = {Senlin Cheng and Haopeng Sun and Tao Xie and Hangyue Zhao and Yiqiang Chen and Bolei Xu and Xiaobo Li},
  title = {{SPT: Sequence Prompt Transformer for Interactive Image Segmentation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=x51A0c6kvg},
}
Figure