SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

Huikai Wu, Junge Zhang, Kaiqi Huang

Abstract

SparseMask is a novel Neural Architecture Search (NAS) framework that automates the design of efficient decoders for dense image prediction tasks such as semantic segmentation, edge detection, and saliency detection. By utilizing a differentiable "sparse loss" function, the framework optimizes connectivity weights within a fully dense search space. SparseMask discovers architectures that are up to 3x faster and use 50% fewer parameters than established models like DeepLab-V3, while maintaining or improving accuracy.

Introduction

Dense image prediction requires assigning a label to every pixel, a task typically handled by encoder-decoder architectures. While encoders are often pre-trained classifiers, decoders are usually manually designed—a process that is both labor-intensive and potentially suboptimal. Traditional NAS methods struggle in this domain due to the high memory demands of high-resolution feature maps and the complexity of multi-scale feature fusion. SparseMask addresses these challenges by treating decoder design as a searchable connectivity problem.

The SparseMask framework operates on a "Fully Dense Network" (FDN) search space where every encoder stage is potentially connected to every decoder stage.

Figure 1: The three-step process of SparseMask: (a) starting with a fully dense network, (b) optimizing continuous connectivity weights using a sparse loss, and (c) pruning low-weight connections to arrive at an efficient architecture.

The search process involves three key components:

  1. Continuous Relaxation: Connection weights ($w$) are relaxed into continuous values between 0 and 1, allowing for gradient-based optimization.
  2. Sparse Loss Function: A specialized loss function encourages both binarization (pushing weights toward 0 or 1) and sparsity (targeting a low average weight $\alpha$).
  3. Architecture Pruning: After training, connections with weights near zero are removed, resulting in a lean, task-specific decoder.

This approach allows the search to complete in approximately 18 GPU-hours, avoiding the high computational costs of reinforcement learning or evolutionary algorithms.

Experimental Results

SparseMask was evaluated on several benchmarks, including PASCAL VOC 2012, ADE20K, MSRA-B, and BSDS500.

Figure 6: Performance comparison on PASCAL VOC. SparseMask (green square) achieves a superior trade-off between accuracy (mIoU) and speed (FPS) compared to random search (blue/yellow circles), MaskConnect (blue triangles), and L1-based pruning (orange triangles).

On the PASCAL VOC dataset, SparseMask achieved a mean Intersection over Union (mIoU) of 73.18%, surpassing DeepLab-V3's 72.51%. Notably, the SparseMask architecture reached 98 FPS, nearly doubling the speed of DeepLab-V3 (50 FPS), while utilizing fewer parameters (0.56M vs 0.67M for the decoder). The results demonstrate that highly specific, sparse connectivity patterns often outperform dense, manually designed alternatives.

Discussion and Conclusion

A key insight from SparseMask is the importance of global context; the search consistently favored connections from the Global Average Pooling (GAP) layer to every decoder stage. Furthermore, architectures optimized for one task (e.g., semantic segmentation) generalized well to others (e.g., edge detection), suggesting the existence of universal optimal connectivity patterns for dense prediction.

While SparseMask currently focuses on connectivity, future work aims to incorporate operation-level search (e.g., kernel sizes) and joint encoder-decoder optimization to further push the boundaries of efficiency in computer vision.

Summary updated

Abstract

In this paper, we aim at automatically searching an efficient network architecture for dense image prediction. Particularly, we follow the encoder-decoder style and focus on designing a connectivity structure for the decoder. To achieve that, we design a densely connected network with learnable connections, named Fully Dense Network, which contains a large set of possible final connectivity structures. We then employ gradient descent to search the optimal connectivity from the dense connections. The search process is guided by a novel loss function, which pushes the weight of each connection to be binary and the connections to be sparse. The discovered connectivity achieves competitive results on two segmentation datasets, while runs more than three times faster and requires less than half parameters compared to the state-of-the-art methods. An extensive experiment shows that the discovered connectivity is compatible with various backbones and generalizes well to other dense image prediction tasks.
BibTeX
@inproceedings{wu2019sparsemask,
  author = {Huikai Wu and Junge Zhang and Kaiqi Huang},
  title = {{SparseMask: Differentiable Connectivity Learning for Dense Image Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00687},
}
Figure