Abstract
SparseMask is a novel Neural Architecture Search (NAS) framework that automates the design of efficient decoders for dense image prediction tasks such as semantic segmentation, edge detection, and saliency detection. By utilizing a differentiable "sparse loss" function, the framework optimizes connectivity weights within a fully dense search space. SparseMask discovers architectures that are up to 3x faster and use 50% fewer parameters than established models like DeepLab-V3, while maintaining or improving accuracy.
Introduction
Dense image prediction requires assigning a label to every pixel, a task typically handled by encoder-decoder architectures. While encoders are often pre-trained classifiers, decoders are usually manually designed—a process that is both labor-intensive and potentially suboptimal. Traditional NAS methods struggle in this domain due to the high memory demands of high-resolution feature maps and the complexity of multi-scale feature fusion. SparseMask addresses these challenges by treating decoder design as a searchable connectivity problem.
Methodology: Differentiable Connectivity Search
The SparseMask framework operates on a "Fully Dense Network" (FDN) search space where every encoder stage is potentially connected to every decoder stage.
The search process involves three key components:
- Continuous Relaxation: Connection weights ($w$) are relaxed into continuous values between 0 and 1, allowing for gradient-based optimization.
- Sparse Loss Function: A specialized loss function encourages both binarization (pushing weights toward 0 or 1) and sparsity (targeting a low average weight $\alpha$).
- Architecture Pruning: After training, connections with weights near zero are removed, resulting in a lean, task-specific decoder.
This approach allows the search to complete in approximately 18 GPU-hours, avoiding the high computational costs of reinforcement learning or evolutionary algorithms.
Experimental Results
SparseMask was evaluated on several benchmarks, including PASCAL VOC 2012, ADE20K, MSRA-B, and BSDS500.
On the PASCAL VOC dataset, SparseMask achieved a mean Intersection over Union (mIoU) of 73.18%, surpassing DeepLab-V3's 72.51%. Notably, the SparseMask architecture reached 98 FPS, nearly doubling the speed of DeepLab-V3 (50 FPS), while utilizing fewer parameters (0.56M vs 0.67M for the decoder). The results demonstrate that highly specific, sparse connectivity patterns often outperform dense, manually designed alternatives.
Discussion and Conclusion
A key insight from SparseMask is the importance of global context; the search consistently favored connections from the Global Average Pooling (GAP) layer to every decoder stage. Furthermore, architectures optimized for one task (e.g., semantic segmentation) generalized well to others (e.g., edge detection), suggesting the existence of universal optimal connectivity patterns for dense prediction.
While SparseMask currently focuses on connectivity, future work aims to incorporate operation-level search (e.g., kernel sizes) and joint encoder-decoder optimization to further push the boundaries of efficiency in computer vision.
Summary updated