Abstract
Dense prediction tasks, such as semantic segmentation and depth estimation, necessitate a balance between high-resolution spatial details and robust semantic context. While the High-Resolution Network (HRNet) maintains high resolution throughout its architecture, it often encounters a semantic bottleneck due to the limited depth of its low-resolution branches. We introduce U-HRNet, a novel backbone that incorporates a U-shaped macro-structure into the HRNet framework. By reallocating computational resources from high-resolution to low-resolution stages and implementing advanced fusion modules, U-HRNet significantly enhances semantic representation without increasing GFLOPs. Our model achieves state-of-the-art performance across multiple benchmarks, including Cityscapes (82.9% mIoU) and ADE20K (47.75% mIoU).
Introduction
Dense prediction requires capturing both fine-grained edges and high-level semantic categories. HRNet addressed this by running multiple resolution branches in parallel. However, two primary limitations persist:
- Semantic Bottleneck: The deepest semantic information from low-resolution branches is not effectively propagated back to high-resolution outputs at the end of the network.
- Depth Constraints: Low-resolution branches lack the depth necessary for complex semantic understanding, as high-resolution convolutions consume the bulk of the computational budget.
The U-HRNet Architecture
U-HRNet addresses these issues by restructuring the parallel branches into a U-shaped macro-structure, relaxing the constraint that all resolutions must be calculated in parallel at every stage.
Key Innovations
- U-Shape Transition: Extra stages are added to act as a decoder, gradually upsampling feature maps and merging them with high-resolution features from earlier encoder stages.
- Computation Reallocation: Recognizing that high-resolution branches primarily capture local textures, U-HRNet reduces their block count. This "saved" budget is moved to low-resolution branches (down to 1/64 scale), allowing for much deeper semantic processing at the same computational cost.
- Advanced Fusion (Fusion B): Instead of simple addition, U-HRNet utilizes a fusion module with pooling and concatenation to enhance cross-scale information flow.
- Cross-Stage Shortcuts: Direct links between early high-resolution features and later upsampling stages preserve spatial precision during deep semantic extraction.
Experimental Evaluation
U-HRNet was rigorously evaluated on diverse datasets including Cityscapes, ADE20K, LIP, and NYUD-V2.
Key Results
- Semantic Segmentation: On Cityscapes, U-HRNet-W48 achieved 81.9% mIoU, outperforming HRNetV2 by 0.8 points. On ADE20K, it set a new benchmark of 47.75% mIoU, a 2-point gain over previous HRNet models.
- Depth Estimation: On NYUD-Large, it reduced RMSE to 0.440, a significant improvement over the HRNet baseline (0.448).
- Medical Imaging: On the Synapse CT dataset, U-HRNet outperformed transformer-based models like TransUNet while remaining purely convolutional and more efficient.
Conclusion
U-HRNet demonstrates that "semantics are cheap while resolution is expensive." By shifting layers to lower resolutions and adopting a U-shaped decoder structure, the network gains significant semantic depth without increasing computational complexity. This architecture proves highly effective for dense prediction tasks, offering a superior alternative to standard parallel high-resolution networks. Future work will explore its integration with specialized task heads for instance segmentation and object detection.
Summary updated