U-HRNet: Delving into Improving Semantic Representation of High Resolution Network for Dense Prediction

Jian Wang, Xiang Long, Guowei Chen, Zewu Wu, Zeyu Chen, Errui Ding

Abstract

Dense prediction tasks, such as semantic segmentation and depth estimation, necessitate a balance between high-resolution spatial details and robust semantic context. While the High-Resolution Network (HRNet) maintains high resolution throughout its architecture, it often encounters a semantic bottleneck due to the limited depth of its low-resolution branches. We introduce U-HRNet, a novel backbone that incorporates a U-shaped macro-structure into the HRNet framework. By reallocating computational resources from high-resolution to low-resolution stages and implementing advanced fusion modules, U-HRNet significantly enhances semantic representation without increasing GFLOPs. Our model achieves state-of-the-art performance across multiple benchmarks, including Cityscapes (82.9% mIoU) and ADE20K (47.75% mIoU).

Introduction

Dense prediction requires capturing both fine-grained edges and high-level semantic categories. HRNet addressed this by running multiple resolution branches in parallel. However, two primary limitations persist:

  1. Semantic Bottleneck: The deepest semantic information from low-resolution branches is not effectively propagated back to high-resolution outputs at the end of the network.
  2. Depth Constraints: Low-resolution branches lack the depth necessary for complex semantic understanding, as high-resolution convolutions consume the bulk of the computational budget.

The U-HRNet Architecture

U-HRNet addresses these issues by restructuring the parallel branches into a U-shaped macro-structure, relaxing the constraint that all resolutions must be calculated in parallel at every stage.

Comparison of (a) HRNet, (b) U-Net, and (c) the proposed U-HRNet architecture.

Key Innovations

  • U-Shape Transition: Extra stages are added to act as a decoder, gradually upsampling feature maps and merging them with high-resolution features from earlier encoder stages.
  • Computation Reallocation: Recognizing that high-resolution branches primarily capture local textures, U-HRNet reduces their block count. This "saved" budget is moved to low-resolution branches (down to 1/64 scale), allowing for much deeper semantic processing at the same computational cost.
  • Advanced Fusion (Fusion B): Instead of simple addition, U-HRNet utilizes a fusion module with pooling and concatenation to enhance cross-scale information flow.
  • Cross-Stage Shortcuts: Direct links between early high-resolution features and later upsampling stages preserve spatial precision during deep semantic extraction.

Experimental Evaluation

U-HRNet was rigorously evaluated on diverse datasets including Cityscapes, ADE20K, LIP, and NYUD-V2.

Qualitative comparison of semantic segmentation results on Cityscapes. U-HRNet shows improved consistency and detail over the standard HRNet.

Key Results

  • Semantic Segmentation: On Cityscapes, U-HRNet-W48 achieved 81.9% mIoU, outperforming HRNetV2 by 0.8 points. On ADE20K, it set a new benchmark of 47.75% mIoU, a 2-point gain over previous HRNet models.
  • Depth Estimation: On NYUD-Large, it reduced RMSE to 0.440, a significant improvement over the HRNet baseline (0.448).
  • Medical Imaging: On the Synapse CT dataset, U-HRNet outperformed transformer-based models like TransUNet while remaining purely convolutional and more efficient.

Conclusion

U-HRNet demonstrates that "semantics are cheap while resolution is expensive." By shifting layers to lower resolutions and adopting a U-shaped decoder structure, the network gains significant semantic depth without increasing computational complexity. This architecture proves highly effective for dense prediction tasks, offering a superior alternative to standard parallel high-resolution networks. Future work will explore its integration with specialized task heads for instance segmentation and object detection.

Summary updated

Abstract

High resolution and advanced semantic representation are both vital for dense prediction. Empirically, low-resolution feature maps often achieve stronger semantic representation, and high-resolution feature maps generally can better identify local features such as edges, but contains weaker semantic information. Existing state-of-the-art frameworks such as HRNet has kept low-resolution and high-resolution feature maps in parallel, and repeatedly exchange the information across different resolutions. However, we believe that the lowest-resolution feature map often contains the strongest semantic information, and it is necessary to go through more layers to merge with high-resolution feature maps, while for high-resolution feature maps, the computational cost of each convolutional layer is very large, and there is no need to go through so many layers. Therefore, we designed a U-shaped High-Resolution Network (U-HRNet), which adds more stages after the feature map with strongest semantic representation and relaxes the constraint in HRNet that all resolutions need to be calculated parallel for a newly added stage. More calculations are allocated to low-resolution feature maps, which significantly improves the overall semantic representation. U-HRNet is a substitute for the HRNet backbone and can achieve significant improvement on multiple semantic segmentation and depth prediction datasets, under the exactly same training and inference setting, with almost no increasing in the amount of calculation. Code is available at PaddleSeg: https://github.com/PaddlePaddle/PaddleSeg.
BibTeX
@misc{wang2022u,
  author = {Jian Wang and Xiang Long and Guowei Chen and Zewu Wu and Zeyu Chen and Errui Ding},
  title = {{U-HRNet: Delving into Improving Semantic Representation of High Resolution Network for Dense Prediction}},
  year = {2022},
  eprint = {2210.07140},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.07140},
}
Figure