Abstract
This paper introduces the Unified Framework for Co-Object Segmentation (UFO), a single Transformer-based architecture designed to solve three distinct computer vision tasks: co-segmentation (CoS), co-saliency detection (CoSD), and video salient object detection (VSOD). Historically, these tasks were addressed with specialized, fragmented models that lacked transferability and efficiency. UFO leverages the self-attention mechanism of Transformers to capture long-range dependencies between images and a novel Intra-MLP module for internal object refinement. The resulting framework achieves state-of-the-art performance across 11 benchmarks while operating at a real-time speed of 140 FPS, demonstrating that a unified approach can be both more general and more efficient than task-specific designs.
Introduction
In computer vision, identifying common or salient objects across groups of images or video frames is critical for tasks like Co-Segmentation, Co-Saliency Detection, and Video Salient Object Detection. Despite their underlying similarities—all involve finding a "common object"—researchers have traditionally treated them as isolated problems. This led to complex, task-specific networks that often relied on computationally expensive similarity matrices or optical flow, resulting in poor real-time performance and limited generalization.
Existing CNN-based methods also struggle with local receptive fields, making it difficult to understand global relationships between distant pixels or across different images in a group. UFO addresses these limitations by replacing task-specific matching strategies with a unified Transformer-based backbone.
Methodology
The UFO framework is built upon a standard encoder (e.g., VGG16) enhanced with two core innovations: Transformer blocks and an Intra-MLP learning module.
Transformer-Based Inter-Image Mining
UFO inserts Transformer blocks into the final layers of the encoder. By treating image features as patch tokens, the self-attention mechanism allows the network to compare every patch in every image of a group simultaneously. This enables the model to "excavate" structured similarities and capture long-range dependencies that traditional CNNs miss. This parallelized approach is significantly faster than previous serial matching algorithms, taking only 5.4ms compared to ~50ms for traditional methods.
Intra-MLP Learning Module
To ensure complete object segmentation and prevent "partial activation" (e.g., highlighting only a bird's head instead of its whole body), the authors introduced the Intra-MLP module. For every pixel, it identifies the top-K most similar pixels within the same image and aggregates their features. This creates a self-mask that helps the decoder produce more accurate and complete object boundaries.
Experimental Results
The framework was rigorously evaluated across 11 benchmarks, including PASCAL-VOC for CoS, CoSOD3k for CoSD, and DAVIS16 for VSOD.
- Efficiency: UFO achieves a blistering 140 FPS on a Titan Xp GPU, making it suitable for real-time applications.
- Accuracy: It outperformed or matched state-of-the-art (SOTA) methods across nearly all benchmarks. On the challenging CoSOD3k dataset, it achieved an S-measure ($S_m$) of 0.819, surpassing specialized models like GCoNet.
- Zero-Prior VSOD: Even without using optical flow—a standard requirement for video tasks—UFO achieved competitive results, proving the strength of its underlying Transformer-based matching logic.
Conclusion
UFO demonstrates that the logic of finding common objects is consistent across static images and video sequences. By unifying these tasks under a single Transformer-based architecture, the authors have created a model that is more general, accurate, and significantly faster than previous specialized solutions. Future work may involve deeper integration of motion cues to further improve robustness in complex video environments with high background clutter.
Summary updated