Abstract
Recent advancements in semantic segmentation have favored increasingly complex, multi-branch architectures to capture multi-scale context. However, these designs often introduce significant hardware latency and optimization challenges. This paper presents FFNet (Fuss-Free Network), a simple yet effective encoder-decoder architecture. By replacing standard bottleneck blocks with basic blocks in a ResNet-style backbone, FFNet achieves a large effective receptive field (ERF) without complex branching. Our results on the Cityscapes dataset demonstrate that FFNet matches the performance of state-of-the-art models like HRNet while offering superior efficiency and hardware friendliness.
Introduction
State-of-the-art (SOTA) semantic segmentation models such as HRNet, FANet, and DDRNet rely on intricate multi-scale fusion mechanisms. While accurate, these architectures are often "hardware-unfriendly," utilizing operations that are difficult to accelerate on standard GPUs and mobile DSPs. A common justification for this complexity is the limited effective receptive field (ERF) of simple classification backbones like standard ResNets. This paper challenges that assumption, asking whether a large receptive field and high accuracy can be achieved through a much simpler, more efficient architecture.
Methodology: The FFNet Architecture
FFNet follows a straightforward Feature Pyramid Network (FPN) structure, designed for both simplicity and performance.
Encoder Design and the Receptive Field
The core innovation lies in the encoder. Standard deep ResNets typically use "bottleneck" blocks to save parameters. However, the authors find that bottleneck blocks "suffocate" the receptive field, which is detrimental to dense prediction tasks. FFNet instead utilizes basic blocks, which significantly enlarge the ERF without requiring dilated convolutions or complex multi-branching.
Flexible Meta-Architecture
The model is designed as a flexible meta-architecture consisting of a Stem, an Encoder, and an "Up-head" Decoder.
- Up-head (Decoder): A lightweight multi-branch head that upsamples and merges features from different stages of the backbone.
- Architectural Variants: The Stem, Up-head, and Task-head can be swapped (configurations A, B, or C) to optimize for specific hardware targets, ranging from high-end GPUs to mobile devices.
- 3-Stage Backbones: For high-resolution inputs, a "3-stage" variant removes the initial high-resolution stage of the backbone to further reduce computational costs.
Experimental Setup
The primary evaluation is conducted on the Cityscapes dataset, focusing on urban street scene segmentation across 19 classes. The models are pre-trained on ImageNet and fine-tuned using the standard 5,000 fine-annotated images, supplemented by 20,000 coarse-annotated images via pseudo-labeling.
Results and Analysis
FFNet demonstrates that a large ERF is the primary driver of success in complex models, and this can be replicated in simpler designs.
Performance and Efficiency
On desktop GPUs, FFNet-GPU-Large (ResNet-150 backbone) achieves 84.4 mIoU on Cityscapes, comparable to HRNetv2-W48 (84.9 mIoU), but with significantly lower latency (195ms vs 249ms on a 1080Ti). As shown in Figure 2, FFNet models consistently define a better Pareto frontier for the latency-accuracy trade-off than both complex models and bottleneck-based variants.
Hardware Friendliness
Because FFNet relies on standard convolutions and simple upsampling, it is highly compatible with standard model acceleration tools. Batch normalization layers can be easily "folded," leading to superior real-world latency. On mobile devices like the Samsung S21, FFNet-Mobile variants achieve real-time performance with mIoU scores between 75.4 and 80.0.
Conclusion
FFNet proves that complex, multi-branch architectures are not a prerequisite for state-of-the-art semantic segmentation. By focusing on the effective receptive field through simple building blocks, FFNet provides a "fuss-free" path to high-performance, hardware-efficient models. Future work includes evaluating these principles on more diverse datasets like ADE20K and exploring Neural Architecture Search (NAS) to further optimize the component configurations.
Summary updated