CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement

Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, Chi-Keung Tang

TL;DR

CascadePSP is a class-agnostic refinement model that transforms coarse, low-resolution segmentation masks into pixel-accurate, high-resolution outputs. By employing a global-to-local cascade strategy, it can handle images up to 4K resolution and beyond without ever having seen high-resolution training data.

What Problem Does This Paper Address?

Modern semantic segmentation models (like DeepLab or PSPNet) are typically trained on fixed-resolution datasets (e.g., PASCAL VOC or COCO). When these models are applied to very high-resolution images (like 4K UHD), their outputs are often blurry and fail to capture fine details along object boundaries. Simply upsampling a low-resolution mask doesn't help because the high-frequency details were never captured in the first place.

Training directly on high-resolution data is difficult for two main reasons:

  1. Memory Constraints: Deep learning models typically use GPU memory that scales linearly with the number of pixels, making 4K training practically impossible on standard hardware.
  2. Data Scarcity: Pixel-accurate annotations for high-resolution images are extremely expensive and time-consuming to produce.

Methods

The authors propose CascadePSP, a general-purpose refinement model designed to be appended to any existing segmentation method. Its core is a Refinement Module (RM) that takes an image and multiple imperfect masks as input.

Key components include:

  • Multi-scale Refinement: The RM generates outputs at different strides (8, 4, and 1). Coarser levels fix overall object structure, while finer levels focus on pixel-level boundary details using skip connections from the backbone.
  • Global-to-Local Cascade:
  • Global Step: The model first processes a downsampled version of the entire image to correct structural errors. Global step architecture
    • Local Step: It then refines details at full resolution by processing overlapping image crops. This "divide-and-conquer" approach bypasses GPU memory limits. Local step architecture
  • Class-Agnostic Training: Instead of training on specific semantic classes, the model is trained to learn "objectness." The authors use a large dataset with over 1,000 classes and train the model to fix masks that have been intentionally "broken" (perturbed) from the ground truth. This allows CascadePSP to generalize to unseen object categories.
  • Gradient Loss: To ensure sharp boundaries, the authors introduce a loss term based on the gradient of the segmentation mask, encouraging it to align perfectly with image edges.
Global step architecture
Local step architecture

Datasets

  • Training: A merged dataset of ~36,500 images from MSRA-10K, DUT-OMRON, ECSSD, and FSS-1000, covering over 1,000 diverse semantic classes.
  • Evaluation:
  • PASCAL VOC 2012: Relabeled 500 images to provide more accurate boundary ground truth.
  • ADE20K: Used for evaluating scene parsing.
  • BIG Dataset: A new high-resolution dataset introduced by the authors, containing 150 objects with resolutions ranging from 2K to 5K.

Key Insights and Ideas

  • Refinement as a Separate Task: By decoupling refinement from initial segmentation, the authors created a tool that can boost the performance of any model without needing to retrain the original.
  • Learning from Perturbation: Training on perturbed ground truth is a clever way to teach a model how to "fix" errors without needing a specific source model's outputs. This makes the system highly robust and truly class-agnostic.
  • Structure vs. Detail: The cascade design acknowledges that fixing a missing limb (structure) requires a different receptive field than sharpening a boundary (detail).
  • Recursive Refinement: The local step can be applied recursively for even higher resolutions, making the method theoretically resolution-independent.

Key Results

CascadePSP achieves significant improvements across all tested datasets and base models.

  • Quantitative: On the high-resolution BIG dataset, it improved the mean Boundary Accuracy (mBA) of DeepLabV3+ from 60.25% to 74.59% and IoU from 89.42% to 92.23%.
  • Qualitative: The visual results are striking. CascadePSP can recover fine details like thin tentacles or complex object edges that are completely lost in standard model outputs. Visual comparison of high-resolution segmentation refinement
Figure 1. Segmentation of a high-resolution image (3492×2328). Left: Produced by Deeplab V3+ [6]. Right: Refined by our algo- rithm.

Limitations and Future Work

  • Semantic Dependency: CascadePSP relies on the initial segmentation for semantic context. If the base model completely misses an object or misclassifies a large region, CascadePSP cannot "invent" the correct semantics; it will simply produce a sharp boundary for the wrong shape.
  • Computational Cost: While it solves the memory problem, the local step involves processing many crops, which adds to the total inference time (e.g., ~6.6 seconds for a high-res image).
  • Future Directions: The authors suggest that this approach could be extended to other high-resolution computer vision tasks beyond segmentation, such as matting or edge detection.

Summary updated

Abstract

State-of-the-art semantic segmentation methods were almost exclusively trained on images within a fixed resolution range. These segmentations are inaccurate for very high-resolution images since using bicubic upsampling of low-resolution segmentation does not adequately capture high-resolution details along object boundaries. In this paper, we propose a novel approach to address the high-resolution segmentation problem without using any high-resolution training data. The key insight is our CascadePSP network which refines and corrects local boundaries whenever possible. Although our network is trained with low-resolution segmentation data, our method is applicable to any resolution even for very high-resolution images larger than 4K. We present quantitative and qualitative studies on different datasets to show that CascadePSP can reveal pixel-accurate segmentation boundaries using our novel refinement module without any finetuning. Thus, our method can be regarded as class-agnostic. Finally, we demonstrate the application of our model to scene parsing in multi-class segmentation.
BibTeX
@inproceedings{cheng2020cascadepsp,
  author = {Ho Kei Cheng and Jihoon Chung and Yu-Wing Tai and Chi-Keung Tang},
  title = {{CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  doi = {10.1109/cvpr42600.2020.00891},
}
Figure