Simple and Efficient Architectures for Semantic Segmentation

Dushyant Mehta, Andrii Skliar, Haitam Ben Yahia, Shubhankar Borse, Fatih Porikli, Amirhossein Habibian, Tijmen Blankevoort

Abstract

Recent advancements in semantic segmentation have favored increasingly complex, multi-branch architectures to capture multi-scale context. However, these designs often introduce significant hardware latency and optimization challenges. This paper presents FFNet (Fuss-Free Network), a simple yet effective encoder-decoder architecture. By replacing standard bottleneck blocks with basic blocks in a ResNet-style backbone, FFNet achieves a large effective receptive field (ERF) without complex branching. Our results on the Cityscapes dataset demonstrate that FFNet matches the performance of state-of-the-art models like HRNet while offering superior efficiency and hardware friendliness.

Introduction

State-of-the-art (SOTA) semantic segmentation models such as HRNet, FANet, and DDRNet rely on intricate multi-scale fusion mechanisms. While accurate, these architectures are often "hardware-unfriendly," utilizing operations that are difficult to accelerate on standard GPUs and mobile DSPs. A common justification for this complexity is the limited effective receptive field (ERF) of simple classification backbones like standard ResNets. This paper challenges that assumption, asking whether a large receptive field and high accuracy can be achieved through a much simpler, more efficient architecture.

Methodology: The FFNet Architecture

FFNet follows a straightforward Feature Pyramid Network (FPN) structure, designed for both simplicity and performance.

Encoder Design and the Receptive Field

The core innovation lies in the encoder. Standard deep ResNets typically use "bottleneck" blocks to save parameters. However, the authors find that bottleneck blocks "suffocate" the receptive field, which is detrimental to dense prediction tasks. FFNet instead utilizes basic blocks, which significantly enlarge the ERF without requiring dilated convolutions or complex multi-branching.

Flexible Meta-Architecture

The model is designed as a flexible meta-architecture consisting of a Stem, an Encoder, and an "Up-head" Decoder.

  1. Up-head (Decoder): A lightweight multi-branch head that upsamples and merges features from different stages of the backbone.
  2. Architectural Variants: The Stem, Up-head, and Task-head can be swapped (configurations A, B, or C) to optimize for specific hardware targets, ranging from high-end GPUs to mobile devices.
  3. 3-Stage Backbones: For high-resolution inputs, a "3-stage" variant removes the initial high-resolution stage of the backbone to further reduce computational costs.
The FFNet meta-architecture consists of a backbone encoder feeding into a compact multi-branch Up-head decoder.

Experimental Setup

The primary evaluation is conducted on the Cityscapes dataset, focusing on urban street scene segmentation across 19 classes. The models are pre-trained on ImageNet and fine-tuned using the standard 5,000 fine-annotated images, supplemented by 20,000 coarse-annotated images via pseudo-labeling.

Results and Analysis

FFNet demonstrates that a large ERF is the primary driver of success in complex models, and this can be replicated in simpler designs.

Performance and Efficiency

On desktop GPUs, FFNet-GPU-Large (ResNet-150 backbone) achieves 84.4 mIoU on Cityscapes, comparable to HRNetv2-W48 (84.9 mIoU), but with significantly lower latency (195ms vs 249ms on a 1080Ti). As shown in Figure 2, FFNet models consistently define a better Pareto frontier for the latency-accuracy trade-off than both complex models and bottleneck-based variants.

Inference latency vs. Cityscapes accuracy on 1080Ti (FP32) and 2080Ti (FP16) shows FFNet models (blue/cyan) matching or exceeding HRNet (red) efficiency.

Hardware Friendliness

Because FFNet relies on standard convolutions and simple upsampling, it is highly compatible with standard model acceleration tools. Batch normalization layers can be easily "folded," leading to superior real-world latency. On mobile devices like the Samsung S21, FFNet-Mobile variants achieve real-time performance with mIoU scores between 75.4 and 80.0.

Conclusion

FFNet proves that complex, multi-branch architectures are not a prerequisite for state-of-the-art semantic segmentation. By focusing on the effective receptive field through simple building blocks, FFNet provides a "fuss-free" path to high-performance, hardware-efficient models. Future work includes evaluating these principles on more diverse datasets like ADE20K and exploring Neural Architecture Search (NAS) to further optimize the component configurations.

Summary updated

Abstract

Though the state-of-the architectures for semantic segmentation, such as HRNet, demonstrate impressive accuracy, the complexity arising from their salient design choices hinders a range of model acceleration tools, and further they make use of operations that are inefficient on current hardware. This paper demonstrates that a simple encoder-decoder architecture with a ResNet-like backbone and a small multi-scale head, performs on-par or better than complex semantic segmentation architectures such as HRNet, FANet and DDRNets.Naïvely applying deep backbones designed for Image Classification to the task of Semantic Segmentation leads to subpar results, owing to a much smaller effective receptive field of these backbones. Implicit among the various design choices put forth in works like HRNet, DDRNet, and FANet are networks with a large effective receptive field. It is natural to ask if a simple encoder-decoder architecture would compare favorably if comprised of backbones that have a larger effective receptive field, though without the use of in-efficient operations like dilated convolutions.We show that with minor and inexpensive modifications to ResNets, enlarging the receptive field, very simple and competitive baselines can be created for Semantic Segmentation. We present a family of such simple architectures for desktop as well as mobile targets, which match or exceed the performance of complex models on the Cityscapes dataset. We hope that our work provides simple yet effective baselines for practitioners to develop efficient semantic segmentation models. The model definitions and pre-trained weights are available at https://github.com/Qualcomm-AI-research/FFNet.
BibTeX
@inproceedings{mehta2022simple,
  author = {Dushyant Mehta and Andrii Skliar and Haitam Ben Yahia and Shubhankar Borse and Fatih Porikli and Amirhossein Habibian and Tijmen Blankevoort},
  title = {{Simple and Efficient Architectures for Semantic Segmentation}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
}
Figure