Abstract
Ultra-high resolution (UHR) image segmentation is critical for applications like satellite imagery and medical scanning but faces significant challenges due to GPU memory constraints and the accuracy-speed trade-off. This paper introduces ISDNet, a bilateral framework that integrates shallow and deep networks to process global context and local details in parallel. By utilizing a Relation-Aware Feature Fusion (RAF) module and high-frequency residuals, ISDNet achieves state-of-the-art performance, running up to 172× faster than previous competitors while maintaining superior precision. Evaluated on benchmarks like DeepGlobe and Inria Aerial, ISDNet demonstrates a new standard for real-time UHR segmentation.
Introduction
Processing ultra-high resolution images (e.g., 4K or 8K) in semantic segmentation typically forces a compromise. Standard deep networks cannot handle such large inputs directly due to memory limits. Existing solutions usually involve either downsampling the image—which sacrifices fine details and small objects—or a "global-local" patching approach. While patching preserves detail, it is computationally expensive because the model must perform dozens of forward passes for a single image.
ISDNet addresses these hurdles by proposing a "best of both worlds" architecture. It processes the whole image directly to maintain global context while efficiently capturing high-resolution details through a lightweight shallow branch. This approach avoids the slow "cropping and merging" pipeline, enabling real-time performance on UHR datasets.
Methods
The core of ISDNet is a bilateral architecture designed to decouple semantic understanding from detail preservation. By splitting these responsibilities, the model avoids redundant computation and optimizes parameter usage.
Bilateral Architecture
The model consists of two primary streams:
- Deep Branch: Processes a downsampled version of the input image to capture high-level global context, such as identifying large-scale features like forests or urban areas.
- Shallow Branch: Instead of standard RGB pixels, this branch takes high-frequency residuals extracted via a Laplacian pyramid. These residuals represent the edges and fine textures of the full-resolution image. Because the network is shallow, it remains computationally efficient even at high resolutions.
Relation-Aware Feature Fusion (RAF)
To combine these disparate features, the authors developed the RAF module. Rather than simple concatenation, RAF uses channel-wise attention and a relationship matrix to intelligently integrate global context with local details, ensuring the final prediction benefits from the strengths of both branches.
Auxiliary Training Tasks
During training, ISDNet employs a Super-Resolution (SR) head and a Structure Distillation loss. These components force the deep branch to learn better spatial representations from low-resolution inputs. Crucially, these modules are discarded during inference, ensuring no impact on the final model's speed.
Results and Discussion
ISDNet was evaluated on three major benchmarks: DeepGlobe (satellite), Inria Aerial (urban), and Cityscapes (autonomous driving).
Performance Benchmarks
On the DeepGlobe dataset (2448 × 2448), ISDNet achieved a 73.30 mIoU at 27.70 FPS. This represents a massive efficiency gain, running 172× faster than the recent FCtL method while delivering higher accuracy. On Inria Aerial (5000 × 5000), it reached 74.23 mIoU at 6.90 FPS, outperforming specialized methods like MagNet and GLNet.
Generalization
The model also showed strong results on Cityscapes, achieving 76.02 mIoU at 50.79 FPS. This demonstrates that the bilateral approach is not limited to aerial imagery but is also effective for real-time street-level segmentation in autonomous driving applications.
Conclusion
ISDNet successfully bridges the gap between accuracy and efficiency for ultra-high resolution segmentation. By leveraging heterogeneous inputs and a novel fusion mechanism, it provides a scalable solution for processing massive images in real-time. Future work may explore adaptive downsampling strategies and automated architecture search for the shallow branch to further optimize performance across diverse lightweight backbones.
Summary updated