SINet: Extreme Lightweight Portrait Segmentation Networks with Spatial Squeeze Module and Information Blocking Decoder

Hyojin Park, Lars Sjosund, Youngjoon Yoo, Nicolas Monet, Jihwan Bang, Nojun KwakWACV 2020

TL;DR

This paper introduces SINet, an extremely lightweight neural network for portrait segmentation that achieves state-of-the-art accuracy with significantly fewer parameters and higher inference speeds. It tackles the challenge of creating efficient segmentation models for real-world applications by proposing an "information blocking decoder" and a "spatial squeeze module" to balance global context and local detail preservation.

What Problem Does This Paper Address?

Portrait segmentation, the task of precisely outlining a person in an image, is crucial for many applications like background editing, virtual try-on, and augmented reality. However, existing segmentation models are often too computationally expensive for real-time use on mobile devices. This is due to the inherent difficulty of segmentation: models need to understand the "big picture" (global consistency) while also capturing fine details like hair strands or clothing edges (local information). Furthermore, publicly available datasets for portrait segmentation are limited in size and diversity, hindering model development.

Methods

SINet addresses these challenges with two main innovations:

  • Information Blocking Decoder: This novel decoder component intelligently filters information from high-resolution feature maps. It uses a confidence score from a lower-resolution map to decide which regions need more detail. If a region is already confidently segmented, the decoder blocks potentially noisy high-resolution information, allowing the model to focus its resources on uncertain areas. This helps prevent errors and improves boundary accuracy.
  • Spatial Squeeze Module (S2-module): To achieve lightweight and efficient multi-receptive field processing (essential for capturing varying object scales), the S2-module uses average pooling to "squeeze" feature map resolutions. This reduces latency compared to traditional multi-path or dilated convolution approaches, which can be computationally heavy and difficult to parallelize on GPUs. The module combines features from multiple receptive fields in an efficient manner.

Additionally, the authors propose a simple data augmentation technique to generate more training data for portrait segmentation, addressing the scarcity of existing datasets.

Datasets

  • EG1800 Dataset: Used for training and evaluating the proposed SINet model. The authors note this dataset is relatively small (around 1,300 training images) and has biases.
  • Baidu Dataset: Used as a source for human images to create augmented training data.
  • Web-scraped Images: Used in conjunction with a heavyweight segmentation model (DeepLabv3+) to generate additional pseudo ground truth segmentation masks.
  • Cityscapes Dataset: Used to demonstrate the generalizability of SINet to broader semantic segmentation tasks beyond portrait segmentation.

Key Insights and Ideas

  • Intelligent Information Filtering is Key: The "information blocking" concept is a clever way to avoid overwhelming the decoder with potentially noisy high-resolution features, leading to more robust segmentation, especially at boundaries.
  • Squeezing for Efficiency: Using average pooling to achieve multi-receptive fields within the S2-module is an effective strategy to reduce computational cost and latency without sacrificing accuracy, a common trade-off in lightweight architectures.
  • Data Augmentation Matters: The proposed method for generating synthetic portrait segmentation data is crucial for improving model performance, especially given the limitations of existing public datasets.
  • Lightweight Doesn't Mean Low Accuracy: SINet demonstrates that it's possible to achieve high accuracy in complex tasks like segmentation with drastically reduced model complexity, opening doors for deployment on resource-constrained devices.

Key Results

  • Extreme Lightweightness: SINet reduces the number of parameters by approximately 95.9% compared to the state-of-the-art PortraitNet (from 2.1M to 86.9K parameters).
  • High Accuracy: Achieves competitive accuracy, with only a slight margin (under 1%) compared to the state-of-the-art method on the EG1800 dataset. SINet+ (using augmented data) achieves 95.29% mIoU.
  • Real-time Performance: Demonstrates impressive inference speeds, achieving 100.6 FPS on an iPhone XS, making it suitable for real-time mobile applications.
  • Generalization: The model shows good performance on the Cityscapes dataset, indicating its potential for general semantic segmentation tasks.
  • Robustness: The information blocking decoder shows improved robustness to image rotations.

Limitations and Future Work

  • Dataset Bias: While the authors propose a data generation method, the original EG1800 dataset still exhibits biases (e.g., in race, age, gender), which could be further addressed.
  • Generalization to Extreme Cases: While tested on Cityscapes, further exploration of SINet's performance on a wider variety of general segmentation tasks could be beneficial.
  • Further Optimization: The authors mention that the S2-module could be further optimized, and future work could explore more advanced techniques for feature extraction and information fusion.

Summary updated

BibTeX
@inproceedings{park2020sinet,
  author = {Hyojin Park and Lars Sjosund and Youngjoon Yoo and Nicolas Monet and Jihwan Bang and Nojun Kwak},
  title = {{SINet: Extreme Lightweight Portrait Segmentation Networks with Spatial Squeeze Module and Information Blocking Decoder}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure