Recurrent Multi-scale Transformer for High-Resolution Salient Object Detection

Xinhao Deng, Pingping Zhang, Wei Liu, Huchuan Lu

Abstract

High-resolution salient object detection (HRSOD) remains challenging due to the limitations of standard models on 4K/8K imagery, which often result in incomplete object detection or blurred boundaries. We present the Recurrent Multi-scale Transformer (RMFormer), which utilizes a recurrent architecture with shared weights to refine predictions across scales consistently. To support this research, we introduce HRS10K, the largest high-resolution SOD dataset to date, containing 10,500 meticulously annotated images at 2K-8K resolution. RMFormer achieves state-of-the-art performance across multiple benchmarks, particularly in boundary precision.

Introduction

Salient Object Detection (SOD) is fundamental for visual tracking and object recognition. While effective at low resolutions, standard models struggle with high-resolution (4K/8K) inputs, producing artifacts like irregular boundaries. Existing high-resolution strategies, such as global-local fusion or multi-stage refinement, often suffer from error accumulation or high parameter counts. Furthermore, the scarcity of large-scale high-resolution datasets has hindered the development of robust models.

RMFormer: Recurrent Multi-scale Architecture

RMFormer addresses these challenges by employing a recurrent refinement strategy rather than a standard feed-forward approach. By sharing weights across scales, the model learns a more robust representation of saliency that is not tied to a specific resolution, preventing the error accumulation common in non-recurrent models.

Comparison of HRSOD strategies: RMFormer (d) uses a recurrent structure for multi-stage refinement.

The architecture consists of several key components:

  1. Coarse Prediction Stage (CPS): Generates an initial saliency map from a low-resolution image using a Swin Transformer backbone.
  2. Recurrent Refinement Stages (RRS): Progressively refines the map at higher scales using the same shared transformer backbone, ensuring consistency and parameter efficiency.
  3. Dual-flow Guided Decoder (DGD): Integrates inter-stage guidance (from lower resolutions) and inner-stage details (from the current scale) to maintain sharp boundaries while filling in object centers.
  4. Pixel-wise Refiner (PR): Targets the most uncertain pixels, typically along edges, for high-precision re-prediction using local and global context.

The HRS10K Dataset

To overcome data limitations, we introduce HRS10K, featuring 10,500 images at 2K-8K resolution. It significantly expands upon previous benchmarks like HRSOD (1,610 images) and UHRSD (4,932 images).

Histogram of edge pixel numbers comparing HRS10K with existing benchmarks.

HRS10K is characterized by:

  • High Complexity: As shown in the edge pixel distribution, it contains objects with significantly higher edge complexity than older datasets.
  • Diversity: It spans 105 categories across 7 major subjects (Stuff, Plant, Person, Object, Mobile, Food, Animal), reducing the performance bias observed in models trained on smaller, less diverse datasets.

Results and Discussion

RMFormer was evaluated against 13 state-of-the-art methods. It consistently ranked at the top across metrics including MAE, F-measure, and mean Boundary Accuracy (mBA). On the HRS10K-TE benchmark, RMFormer achieved an mBA of 0.766, surpassing the previous best (PGNet) at 0.745. Qualitative results demonstrate its superior ability to capture fine details, such as fluffy fur or thin structures like Ferris wheel wires.

While highly accurate, the model is computationally intensive, requiring 563.14 GFLOPs and running at approximately 16.1 FPS on an RTX 3090. This makes it slower than some low-resolution-only models, though significantly more precise for high-resolution applications.

Conclusion

RMFormer provides a robust solution for high-resolution SOD through recurrent refinement and targeted pixel-wise processing. By introducing the HRS10K dataset, we provide a more challenging and diverse benchmark for the community. Future work will focus on optimizing the architecture to reduce computational overhead and improve inference speed for real-time high-resolution applications.

Summary updated

BibTeX
@inproceedings{deng2023recurrent,
  author = {Xinhao Deng and Pingping Zhang and Wei Liu and Huchuan Lu},
  title = {{Recurrent Multi-scale Transformer for High-Resolution Salient Object Detection}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611983},
}

Loading related content...

Figure