Explicit Shape Encoding for Real-Time Instance Segmentation

Wenqiang Xu, Haiyang Wang, Fubo Qi, Cewu Lu

Abstract: In this paper, we propose a novel top-down instance segmentation framework based on explicit shape encoding, named ESE-Seg. It largely reduces the computational consumption of instance segmentation by explicitly decoding multiple object shapes with tensor operations, performing at almost the same speed as object detection. ESE-Seg utilizes a novel shape signature called Inner-center Radius (IR), Chebyshev polynomial fitting, and modern object detectors. ESE-Seg with YOLOv3 outperforms Mask R-CNN on Pascal VOC 2012 at $mAP^[email protected]$ while being 7 times faster.

Introduction

Instance segmentation—identifying both the location and the specific pixels of every object in an image—is traditionally significantly slower than object detection. While a detector outputs a simple 4D bounding box vector, models like Mask R-CNN rely on "implicit" encoding, generating high-resolution binary masks through heavy upsampling layers and multiple decoder passes. This computational bottleneck hinders real-time applications in autonomous driving and robotics. ESE-Seg addresses this by representing complex shapes as "explicit" short vectors that a network can regress as easily as a bounding box.

Methodology

The ESE-Seg framework introduces three key technical innovations to achieve high-speed segmentation:

  1. Inner-center Radius (IR) Shape Signature: Instead of predicting a 2D mask, the model identifies an "inner-center" (the point furthest from the object's boundary) and samples the distance to the contour at various angles ($\theta$). This transforms a 2D shape into a 1D functional representation: $f(\theta)$.
  2. Chebyshev Polynomial Fitting: To compress the shape signature into a robust, low-dimensional vector, the authors employ Chebyshev polynomials. Unlike standard polynomials or Fourier transforms, Chebyshev polynomials are numerically stable and avoid "Runge’s phenomenon" (oscillations at interval edges), making them ideal for neural network training.
  3. Explicit Decoding: The shape is represented by a mathematical formula, allowing the "decoder" to consist of simple tensor operations (multiplication and addition) rather than a neural network. This enables simultaneous reconstruction of all object shapes in a single pass.
The ESE-Seg architecture: (a) CNN encoding into a shape vector and bounding box/class, followed by (b) parallelized decoding of the shape.

ESE-Seg is detector-agnostic and can be integrated into architectures like YOLOv3, Faster R-CNN, or RetinaNet.

Experimental Evaluation

The authors evaluated ESE-Seg on three standard benchmarks: Pascal VOC 2012, Pascal SBD 2012, and the large-scale COCO 2017 dataset. Key insights from the evaluation include:

  • 1D Stability: Regressing a 1D sequence (radius at angles) is more stable for CNNs than regressing 2D vertex coordinates ($x, y$), which are highly noise-sensitive.
  • Numerical Stability: Chebyshev coefficients provide a better numerical distribution for training compared to Fourier series, which are sensitive to frequency parameters and can lead to unstable gradients.
  • Resolution Independence: Because shapes are encoded explicitly, output contour resolution can be increased at inference time by sampling more points from the polynomial without increasing network workload.

Results

ESE-Seg demonstrates a significant leap in efficiency without sacrificing competitive accuracy at standard thresholds.

Qualitative results of ESE-Seg on Pascal VOC and COCO datasets, showing successful instance segmentation across various object categories.
  • Speed and Efficiency: ESE-Seg with a YOLOv3 backbone is 7 times faster than Mask R-CNN on Pascal VOC 2012. Using a YOLOv3-tiny backbone, the system achieves 130 FPS on a GTX 1080Ti.
  • Accuracy: On Pascal VOC 2012, ESE-Seg achieved a $mAP^[email protected]$ of 69.3, surpassing Mask R-CNN's 68.5. On the complex COCO dataset, it remains competitive with other real-time methods.

Conclusion

ESE-Seg offers a high-speed alternative for instance segmentation by treating object shapes as regressible vectors. While the method excels at real-time performance, the compression into short vectors (e.g., 20 coefficients) results in a loss of fine-grained detail for highly complex or "thin" structures. Future work will explore representations that capture high-frequency details while maintaining the efficiency of explicit encoding.

Summary updated

Abstract

In this paper, we propose a novel top-down instance segmentation framework based on explicit shape encoding, named \textbf{ESE-Seg}. It largely reduces the computational consumption of the instance segmentation by explicitly decoding the multiple object shapes with tensor operations, thus performs the instance segmentation at almost the same speed as the object detection. ESE-Seg is based on a novel shape signature Inner-center Radius (IR), Chebyshev polynomial fitting and the strong modern object detectors. ESE-Seg with YOLOv3 outperforms the Mask R-CNN on Pascal VOC 2012 at mAP <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</sup> @0.5 while 7 times faster.
BibTeX
@inproceedings{xu2019explicit,
  author = {Wenqiang Xu and Haiyang Wang and Fubo Qi and Cewu Lu},
  title = {{Explicit Shape Encoding for Real-Time Instance Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=WqIkmj2JNbV},
}
Figure