Squeezing Context into Patches: Towards Memory-Efficient Ultra-High Resolution Semantic Segmentation

Wang Liu, Puhong Duan, Xudong Kang, Shutao LiIJCAI 2025 · DOI

TL;DR

This paper introduces SCPSeg, a novel single-branch semantic segmentation method for ultra-high-resolution (UHR) images that efficiently compresses global context into local patches. By balancing detailed local information with broad contextual understanding without increasing memory usage during inference, SCPSeg achieves state-of-the-art accuracy on several UHR segmentation benchmarks.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, is crucial for many applications like remote sensing and autonomous driving. However, segmenting ultra-high-resolution (UHR) images presents a significant challenge due to memory constraints on GPUs. Existing methods often resort to downscaling images (losing detail) or using multi-branch architectures (increasing memory). This creates a trade-off between capturing fine-grained local details and understanding the broader global context, leading to either inaccurate segmentations or prohibitive memory requirements, especially for deployment on edge devices.

Methods

SCPSeg tackles this challenge with a novel single-branch architecture that integrates context compression and local feature alignment. The core innovations are:

  • Context Squeezing Module (CSM): This module compresses redundant global context information from surrounding areas into local patches. It achieves this by resizing a local patch, then stacking and concatenating downscaled versions of the surrounding context with the resized local patch. This effectively embeds global context into the local patch representation without losing critical details.
  • Super-Resolution Guided Local Feature Alignment (LFA): To further enhance detail preservation, an auxiliary super-resolution decoder is employed. This decoder learns to reconstruct high-resolution local features. The LFA then aligns these super-resolution features with the segmentation decoder's features. This alignment is achieved by minimizing the similarity matrix differences between sliding windows in the two feature maps, effectively transferring structural information and improving edge precision.
  • Single-Branch Encoder: Unlike many UHR methods that use multiple branches, SCPSeg employs a single-branch encoder that processes UHR images in a sliding window manner. This significantly reduces memory overhead during inference.
  • Multi-Task Learning: The model is trained using a combined loss function that includes cross-entropy loss for segmentation, a super-resolution loss, and the proposed local feature alignment loss.

Datasets

The authors evaluate SCPSeg on four widely used UHR semantic segmentation benchmarks:

  • ISPRS Potsdam: An urban land cover mapping dataset with 38 tiles of size 6000x6000.
  • BLU: A dataset covering urban and rural areas, with original tiles of size 15680x15680, cropped to 2048x2048 for processing.
  • DeepGlobe: A land cover mapping dataset with 803 tiles of size 2448x2448.
  • Inria Aerial: A building extraction dataset featuring large images (5000x5000 pixels), approximately four times the size of DeepGlobe images.

Key Insights and Ideas

  • Context Compression is Key: The CSM's ability to "squeeze" global context into local patches is crucial. It demonstrates that you don't need a separate, high-memory global branch to understand the broader scene; this information can be effectively encoded locally.
  • Super-Resolution as a Feature Enhancer: Using a super-resolution decoder not just for its primary task but as a source of rich structural information to guide local feature alignment is an elegant way to improve segmentation precision.
  • Single-Branch Efficiency: The success of a single-branch architecture for UHR segmentation is a significant finding, directly addressing the memory bottleneck without sacrificing accuracy. This opens doors for more efficient UHR processing.
  • Balancing Context and Detail: The paper highlights the delicate balance needed between context window size and down-scaled local patch size. Too much context can lead to degradation, while too little limits understanding.

Key Results

SCPSeg consistently achieves state-of-the-art performance across the evaluated datasets, outperforming both generic and specialized UHR segmentation methods in terms of accuracy metrics like mIoU and mF1. Crucially, it does so while maintaining significantly lower GPU memory usage during inference compared to other UHR methods. For instance, on the ISPRS Potsdam dataset, SCPSeg achieved the highest accuracy and the lowest memory footprint among all tested models. On the Inria Aerial dataset, it significantly outperformed other UHRSS methods in accuracy while maintaining superior memory efficiency.

Limitations and Future Work

The authors acknowledge that the performance gains are influenced by the choice of context window size and down-scaled local size, suggesting that finding the optimal balance is important. They also note that while their method is memory-efficient during inference, the training process still involves some computational cost.

Future work includes extending SCPSeg to other high-resolution vision tasks like object detection and instance segmentation. They also plan to investigate adaptive mechanisms to dynamically adjust memory usage based on image complexity, further enhancing its applicability in diverse scenarios.

Summary updated

BibTeX
@inproceedings{liu2025squeezing,
  author = {Wang Liu and Puhong Duan and Xudong Kang and Shutao Li},
  title = {{Squeezing Context into Patches: Towards Memory-Efficient Ultra-High Resolution Semantic Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/179},
}
Figure