Abstract: In this paper, we propose a novel top-down instance segmentation framework based on explicit shape encoding, named ESE-Seg. It largely reduces the computational consumption of instance segmentation by explicitly decoding multiple object shapes with tensor operations, performing at almost the same speed as object detection. ESE-Seg utilizes a novel shape signature called Inner-center Radius (IR), Chebyshev polynomial fitting, and modern object detectors. ESE-Seg with YOLOv3 outperforms Mask R-CNN on Pascal VOC 2012 at $mAP^[email protected]$ while being 7 times faster.
Introduction
Instance segmentation—identifying both the location and the specific pixels of every object in an image—is traditionally significantly slower than object detection. While a detector outputs a simple 4D bounding box vector, models like Mask R-CNN rely on "implicit" encoding, generating high-resolution binary masks through heavy upsampling layers and multiple decoder passes. This computational bottleneck hinders real-time applications in autonomous driving and robotics. ESE-Seg addresses this by representing complex shapes as "explicit" short vectors that a network can regress as easily as a bounding box.
Methodology
The ESE-Seg framework introduces three key technical innovations to achieve high-speed segmentation:
- Inner-center Radius (IR) Shape Signature: Instead of predicting a 2D mask, the model identifies an "inner-center" (the point furthest from the object's boundary) and samples the distance to the contour at various angles ($\theta$). This transforms a 2D shape into a 1D functional representation: $f(\theta)$.
- Chebyshev Polynomial Fitting: To compress the shape signature into a robust, low-dimensional vector, the authors employ Chebyshev polynomials. Unlike standard polynomials or Fourier transforms, Chebyshev polynomials are numerically stable and avoid "Runge’s phenomenon" (oscillations at interval edges), making them ideal for neural network training.
- Explicit Decoding: The shape is represented by a mathematical formula, allowing the "decoder" to consist of simple tensor operations (multiplication and addition) rather than a neural network. This enables simultaneous reconstruction of all object shapes in a single pass.
ESE-Seg is detector-agnostic and can be integrated into architectures like YOLOv3, Faster R-CNN, or RetinaNet.
Experimental Evaluation
The authors evaluated ESE-Seg on three standard benchmarks: Pascal VOC 2012, Pascal SBD 2012, and the large-scale COCO 2017 dataset. Key insights from the evaluation include:
- 1D Stability: Regressing a 1D sequence (radius at angles) is more stable for CNNs than regressing 2D vertex coordinates ($x, y$), which are highly noise-sensitive.
- Numerical Stability: Chebyshev coefficients provide a better numerical distribution for training compared to Fourier series, which are sensitive to frequency parameters and can lead to unstable gradients.
- Resolution Independence: Because shapes are encoded explicitly, output contour resolution can be increased at inference time by sampling more points from the polynomial without increasing network workload.
Results
ESE-Seg demonstrates a significant leap in efficiency without sacrificing competitive accuracy at standard thresholds.
- Speed and Efficiency: ESE-Seg with a YOLOv3 backbone is 7 times faster than Mask R-CNN on Pascal VOC 2012. Using a YOLOv3-tiny backbone, the system achieves 130 FPS on a GTX 1080Ti.
- Accuracy: On Pascal VOC 2012, ESE-Seg achieved a $mAP^[email protected]$ of 69.3, surpassing Mask R-CNN's 68.5. On the complex COCO dataset, it remains competitive with other real-time methods.
Conclusion
ESE-Seg offers a high-speed alternative for instance segmentation by treating object shapes as regressible vectors. While the method excels at real-time performance, the compression into short vectors (e.g., 20 coefficients) results in a loss of fine-grained detail for highly complex or "thin" structures. Future work will explore representations that capture high-frequency details while maintaining the efficiency of explicit encoding.
Summary updated