Sempart: Self-supervised Multi-resolution Partitioning of Image Semantics

Sriram Ravindran, Debraj BasuICCV 2023 · DOI · arXiv

Introduction

Identifying the most important or "salient" object in an image is a foundational task in computer vision. Traditionally, training models for saliency detection has required massive amounts of hand-annotated data. While recent self-supervised models like DINO have demonstrated an inherent ability to understand image semantics without human labels, the masks they produce are often low-resolution and "blobby," lacking the sharp edges and fine details of real-world objects.

To address this, researchers typically rely on slow, iterative post-processing techniques like Bilateral Filtering or Conditional Random Fields (CRFs) to align blurry masks with image boundaries. This creates a trade-off: one must choose between fast, low-quality masks or slow, high-quality ones. SEMPART (Self-supervised Multi-resolution Partitioning) aims to bridge this gap, providing high-fidelity, edge-aware masks in a single, rapid inference pass.

The SEMPART Architecture

SEMPART introduces a dual-branch architecture built on top of a frozen DINO backbone. By processing information at multiple resolutions simultaneously, the model learns to balance global semantic consistency with local structural detail.

The SEMPART architecture featuring co-optimized coarse and fine branches.

Dual-Branch Methodology

  • The Coarse Branch: Operating at a low resolution (40x40), this branch employs a Normalized Cut (Ncut) loss. This graph-theoretic approach treats image patches as nodes and seeks the most natural "split" between foreground and background based on semantic similarity.
  • The Fine Branch: This convolutional refinement network progressively upsamples low-resolution features. It uses the original high-resolution RGB image as a guide to ensure the resulting mask respects physical boundaries.
  • Graph Total Variation (GTV): To ensure the fine mask follows object boundaries accurately, the authors use GTV regularization. This constraint encourages the mask to be smooth where image colors are similar and allows for sharp transitions only where the image itself has edges.

Key Insights: Co-Optimization and Efficiency

A critical innovation in SEMPART is the joint optimization of these branches. The coarse branch acts as a semantic anchor, preventing the fine branch from becoming distracted by high-frequency texture details. This co-optimization allows the shared transformer encoder to develop a better semantic foundation.

Furthermore, by baking the refinement process directly into the neural network architecture rather than treating it as a post-processing step, SEMPART eliminates the need for iterative refinement. This architectural choice results in a significant speed advantage, with the model clocking in at approximately 14ms per image—nearly 24x faster than methods requiring traditional Bilateral Filtering.

Performance and Results

SEMPART was evaluated across several standard benchmarks, including DUT-OMRON, DUTS-TE, and ECSSD for saliency detection, as well as COCO and VOC for object localization.

Qualitative comparison of SEMPART against other self-supervised methods like TokenCut and MOVE across various image types.

As shown in the qualitative comparisons, SEMPART-fine consistently produces masks that are more aligned with ground truth than previous state-of-the-art methods like MOVE or TokenCut. Quantitatively, SEMPART outperformed MOVE by an average of 3.7% in max Fβ and 2.7% in IoU across saliency datasets. In object localization tasks, the model produced high-quality bounding boxes competitive with or superior to existing self-supervised methods.

Limitations and Future Work

Despite its success, SEMPART inherits certain biases from its DINO backbone, which was pre-trained on ImageNet. This often results in a "human/animal bias," where the model prioritizes living subjects over static objects like furniture, even if the latter is the intended salient object. Additionally, the Graph Total Variation (GTV) loss can occasionally be too aggressive, smoothing over extremely thin structures or merging narrow adjacent regions.

Future research may explore applying this multi-resolution approach to more complex tasks, such as class-aware object detection or improving the precision of visual search engines.

Summary updated

BibTeX
@inproceedings{ravindran2023sempart,
  author = {Sriram Ravindran and Debraj Basu},
  title = {{Sempart: Self-supervised Multi-resolution Partitioning of Image Semantics}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00073},
}
Figure