A Unified Transformer Framework for Group-Based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection

Yukun Su, Jingliang Deng, Ruizhou Sun, Guosheng Lin, Hanjing Su, Qingyao Wu

Abstract

This paper introduces the Unified Framework for Co-Object Segmentation (UFO), a single Transformer-based architecture designed to solve three distinct computer vision tasks: co-segmentation (CoS), co-saliency detection (CoSD), and video salient object detection (VSOD). Historically, these tasks were addressed with specialized, fragmented models that lacked transferability and efficiency. UFO leverages the self-attention mechanism of Transformers to capture long-range dependencies between images and a novel Intra-MLP module for internal object refinement. The resulting framework achieves state-of-the-art performance across 11 benchmarks while operating at a real-time speed of 140 FPS, demonstrating that a unified approach can be both more general and more efficient than task-specific designs.

Introduction

In computer vision, identifying common or salient objects across groups of images or video frames is critical for tasks like Co-Segmentation, Co-Saliency Detection, and Video Salient Object Detection. Despite their underlying similarities—all involve finding a "common object"—researchers have traditionally treated them as isolated problems. This led to complex, task-specific networks that often relied on computationally expensive similarity matrices or optical flow, resulting in poor real-time performance and limited generalization.

The UFO framework shifts from task-specific models to a unified architecture capable of handling CoS, CoSD, and VSOD simultaneously.

Existing CNN-based methods also struggle with local receptive fields, making it difficult to understand global relationships between distant pixels or across different images in a group. UFO addresses these limitations by replacing task-specific matching strategies with a unified Transformer-based backbone.

Methodology

The UFO framework is built upon a standard encoder (e.g., VGG16) enhanced with two core innovations: Transformer blocks and an Intra-MLP learning module.

Transformer-Based Inter-Image Mining

UFO inserts Transformer blocks into the final layers of the encoder. By treating image features as patch tokens, the self-attention mechanism allows the network to compare every patch in every image of a group simultaneously. This enables the model to "excavate" structured similarities and capture long-range dependencies that traditional CNNs miss. This parallelized approach is significantly faster than previous serial matching algorithms, taking only 5.4ms compared to ~50ms for traditional methods.

The UFO architecture integrates Transformer blocks for inter-image communication and an Intra-MLP module for internal feature refinement.

Intra-MLP Learning Module

To ensure complete object segmentation and prevent "partial activation" (e.g., highlighting only a bird's head instead of its whole body), the authors introduced the Intra-MLP module. For every pixel, it identifies the top-K most similar pixels within the same image and aggregates their features. This creates a self-mask that helps the decoder produce more accurate and complete object boundaries.

Experimental Results

The framework was rigorously evaluated across 11 benchmarks, including PASCAL-VOC for CoS, CoSOD3k for CoSD, and DAVIS16 for VSOD.

  • Efficiency: UFO achieves a blistering 140 FPS on a Titan Xp GPU, making it suitable for real-time applications.
  • Accuracy: It outperformed or matched state-of-the-art (SOTA) methods across nearly all benchmarks. On the challenging CoSOD3k dataset, it achieved an S-measure ($S_m$) of 0.819, surpassing specialized models like GCoNet.
  • Zero-Prior VSOD: Even without using optical flow—a standard requirement for video tasks—UFO achieved competitive results, proving the strength of its underlying Transformer-based matching logic.

Conclusion

UFO demonstrates that the logic of finding common objects is consistent across static images and video sequences. By unifying these tasks under a single Transformer-based architecture, the authors have created a model that is more general, accurate, and significantly faster than previous specialized solutions. Future work may involve deeper integration of motion cues to further improve robustness in complex video environments with high background clutter.

Summary updated

Abstract

Humans tend to mine objects by learning from a group of images or several frames of video since we live in a dynamic world. In the computer vision area, many researches focus on co-segmentation (CoS), co-saliency detection (CoSD) and video salient object detection (VSOD) to discover the co-occurrent objects. However, previous approaches design different networks on these similar tasks separately, and they are difficult to apply to each other, which lowers the upper bound of the transferability of deep learning frameworks. Besides, they fail to take full advantage of the cues among inter- and intra-feature within a group of images. In this paper, we introduce a unified framework to tackle these issues, term as UFO (Unified Framework for Co-Object Segmentation). Specifically, we first introduce a transformer block, which views the image feature as a patch token and then captures their long-range dependencies through the self-attention mechanism. This can help the network to excavate the patch structured similarities among the relevant objects. Furthermore, we propose an intra-MLP learning module to produce self-mask to enhance the network to avoid partial activation. Extensive experiments on four CoS benchmarks (PASCAL, iCoseg, Internet and MSRC), three CoSD benchmarks (Cosal2015, CoSOD3k, and CocA) and four VSOD benchmarks (DAVIS16, FBMS, ViSal and SegV2) show that our method outperforms other state-of-the-arts on three different tasks in both accuracy and speed by using the same network architecture , which can reach 140 FPS in real-time.
BibTeX
@article{su2022unified,
  author = {Yukun Su and Jingliang Deng and Ruizhou Sun and Guosheng Lin and Hanjing Su and Qingyao Wu},
  title = {{A Unified Transformer Framework for Group-Based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
}
Figure