ISDNet: Integrating Shallow and Deep Networks for Efficient Ultra-high Resolution Segmentation

Shaohua Guo, Liang Liu, Zhenye Gan, Yabiao Wang, Wuhao Zhang, Chengjie Wang, Guannan Jiang, Wei Zhang, Ran Yi, Lizhuang Ma, Ke Xu

Abstract

Ultra-high resolution (UHR) image segmentation is critical for applications like satellite imagery and medical scanning but faces significant challenges due to GPU memory constraints and the accuracy-speed trade-off. This paper introduces ISDNet, a bilateral framework that integrates shallow and deep networks to process global context and local details in parallel. By utilizing a Relation-Aware Feature Fusion (RAF) module and high-frequency residuals, ISDNet achieves state-of-the-art performance, running up to 172× faster than previous competitors while maintaining superior precision. Evaluated on benchmarks like DeepGlobe and Inria Aerial, ISDNet demonstrates a new standard for real-time UHR segmentation.

Introduction

Processing ultra-high resolution images (e.g., 4K or 8K) in semantic segmentation typically forces a compromise. Standard deep networks cannot handle such large inputs directly due to memory limits. Existing solutions usually involve either downsampling the image—which sacrifices fine details and small objects—or a "global-local" patching approach. While patching preserves detail, it is computationally expensive because the model must perform dozens of forward passes for a single image.

ISDNet addresses these hurdles by proposing a "best of both worlds" architecture. It processes the whole image directly to maintain global context while efficiently capturing high-resolution details through a lightweight shallow branch. This approach avoids the slow "cropping and merging" pipeline, enabling real-time performance on UHR datasets.

Methods

The core of ISDNet is a bilateral architecture designed to decouple semantic understanding from detail preservation. By splitting these responsibilities, the model avoids redundant computation and optimizes parameter usage.

Bilateral Architecture

The model consists of two primary streams:

  1. Deep Branch: Processes a downsampled version of the input image to capture high-level global context, such as identifying large-scale features like forests or urban areas.
  2. Shallow Branch: Instead of standard RGB pixels, this branch takes high-frequency residuals extracted via a Laplacian pyramid. These residuals represent the edges and fine textures of the full-resolution image. Because the network is shallow, it remains computationally efficient even at high resolutions.
The ISDNet architecture featuring bilateral branches and Relation-Aware Feature Fusion.

Relation-Aware Feature Fusion (RAF)

To combine these disparate features, the authors developed the RAF module. Rather than simple concatenation, RAF uses channel-wise attention and a relationship matrix to intelligently integrate global context with local details, ensuring the final prediction benefits from the strengths of both branches.

Auxiliary Training Tasks

During training, ISDNet employs a Super-Resolution (SR) head and a Structure Distillation loss. These components force the deep branch to learn better spatial representations from low-resolution inputs. Crucially, these modules are discarded during inference, ensuring no impact on the final model's speed.

Results and Discussion

ISDNet was evaluated on three major benchmarks: DeepGlobe (satellite), Inria Aerial (urban), and Cityscapes (autonomous driving).

Performance Benchmarks

On the DeepGlobe dataset (2448 × 2448), ISDNet achieved a 73.30 mIoU at 27.70 FPS. This represents a massive efficiency gain, running 172× faster than the recent FCtL method while delivering higher accuracy. On Inria Aerial (5000 × 5000), it reached 74.23 mIoU at 6.90 FPS, outperforming specialized methods like MagNet and GLNet.

Qualitative and quantitative comparison of ISDNet against baseline and state-of-the-art models.

Generalization

The model also showed strong results on Cityscapes, achieving 76.02 mIoU at 50.79 FPS. This demonstrates that the bilateral approach is not limited to aerial imagery but is also effective for real-time street-level segmentation in autonomous driving applications.

Conclusion

ISDNet successfully bridges the gap between accuracy and efficiency for ultra-high resolution segmentation. By leveraging heterogeneous inputs and a novel fusion mechanism, it provides a scalable solution for processing massive images in real-time. Future work may explore adaptive downsampling strategies and automated architecture search for the shallow branch to further optimize performance across diverse lightweight backbones.

Summary updated

Abstract

The huge burden of computation and memory are two obstacles in ultra-high resolution image segmentation. To tackle these issues, most of the previous works follow the global-local refinement pipeline, which pays more attention to the memory consumption but neglects the inference speed. In comparison to the pipeline that partitions the large image into small local regions, we focus on inferring the whole image directly. In this paper, we propose ISDNet, a novel ultra-high resolution segmentation framework that integrates the shallow and deep networks in a new manner, which significantly accelerates the inference speed while achieving accurate segmentation. To further exploit the relationship between the shallow and deep features, we propose a novel Relational-Aware feature Fusion module, which ensures high performance and robustness of our framework. Extensive experiments on Deepglobe, Inria Aerial, and Cityscapes datasets demonstrate our performance is consistently superior to state-of-the-arts. Specifically, it achieves 73.30 mIoU with a speed of 27.70 FPS on Deepglobe, which is more accurate and 172 × faster than the recent competitor. Code available at https://github.com/cedricgsh/ISDNet.
BibTeX
@inproceedings{guo2022isdnet,
  author = {Shaohua Guo and Liang Liu and Zhenye Gan and Yabiao Wang and Wuhao Zhang and Chengjie Wang and Guannan Jiang and Wei Zhang and Ran Yi and Lizhuang Ma and Ke Xu},
  title = {{ISDNet: Integrating Shallow and Deep Networks for Efficient Ultra-high Resolution Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00432},
}

Loading related content...

Figure