ContourRend: A Segmentation Method for Improving Contours by Rendering

Junwen Chen, Yi Lu, Yaran Chen, Dongbin Zhao, Zhonghua Pang

Abstract

ContourRend is a novel image segmentation framework designed to overcome the limitations of traditional mask-based and contour-based methods. By combining a Graph Convolutional Network (GCN) for initial boundary estimation with a high-resolution MLP-based rendering step, the framework significantly reduces edge blurriness and captures fine geometric details. Evaluated on the Cityscapes dataset, ContourRend demonstrates superior performance in boundary accuracy and mean Intersection over Union (mIoU).

Introduction

In computer vision, image segmentation typically follows one of two paradigms: mask-based methods or contour-based methods. Mask-based approaches, such as Mask R-CNN, predict labels on a coarse grid, which often results in "blocky" or blurry edges when upsampled. Contour-based methods, like Curve-GCN, predict vertex coordinates along an object's boundary. While these provide sharp lines, they are often constrained by a fixed number of vertices, causing them to miss complex details like the spokes of a wheel or the fine curves of clothing.

ContourRend addresses this "detail gap" by introducing a hybrid architecture that generates a rough contour and then refines it through a specialized rendering process.

Methodology

The ContourRend architecture consists of two primary components: a Contour Generator and a Contour Renderer.

Overall architecture of the ContourRend framework, illustrating the pipeline from input image to the final refined mask.

Contour Generator

The first stage utilizes a Graph Convolutional Network (GCN) built upon a DeepLab-ResNet backbone. This generator treats the object boundary as a ring of nodes and employs GCN layers to iteratively regress the coordinates of these vertices. This process produces an initial "rough draft" of the object's shape, providing a geometric foundation for the subsequent refinement stage.

Detailed architecture of the GCN-based Contour Generator, showing the integration of edge and vertex branches to adjust the initial contour.

Contour Renderer

The core innovation of this framework is the Contour Renderer. Inspired by the PointRend concept, this module samples points in the immediate vicinity of the predicted contour vertices.

  • Training Phase: The system randomly offsets vertex coordinates to sample points near the boundary.
  • Inference Phase: A dense grid is created around each vertex to ensure high-resolution coverage.

A lightweight Multi-Layer Perceptron (MLP) then processes the features of these sampled points to predict their status as foreground or background. These high-resolution predictions are "pasted" back onto the initial mask, resulting in a sharp and detailed final boundary.

Datasets and Evaluation

The researchers evaluated ContourRend using the Cityscapes dataset, focusing on single-object segmentation. The dataset was partitioned into 45,984 training, 3,936 validation, and 9,784 test images. The task involved segmenting centered objects from common urban categories, including Person, Bicycle, Car, Bus, and Truck, using $224 \times 224$ input images.

Key Results and Insights

ContourRend achieved a mean IoU of 72.41%, representing a 1.22% improvement over the Polygon-GCN baseline. Key findings include:

  • Boundary Focus: By concentrating computational resources on pixels near the contour rather than the object's interior, the model achieves higher efficiency and detail.
  • Auxiliary Loss: The renderer acts as an auxiliary loss during training, which helps the GCN-based generator learn more robust features. Ablation studies showed a 7.16% improvement in mIoU for the full system compared to a standalone generator.
  • Qualitative Superiority: Visual analysis confirms that ContourRend successfully captures intricate details, such as the specific shape of a handbag or the curvature of a tire, which are often lost in standard contour models.

Conclusion and Future Work

ContourRend effectively bridges the gap between coarse mask predictions and rigid contour models. While currently optimized for single-object segmentation with a fixed vertex count, the modular nature of the rendering mechanism suggests it could be integrated into other architectures like PolarMask. Future research will focus on extending this approach to multi-object instance segmentation in complex, crowded environments.

Summary updated

Abstract

A good object segmentation should contain clear contours and complete regions. However, mask-based segmentation can not handle contour features well on a coarse prediction grid, thus causing problems of blurry edges. While contour-based segmentation provides contours directly, but misses contours’ details. In order to obtain fine contours, we propose a segmentation method named ContourRend which adopts a contour renderer to refine segmentation contours. And we implement our method on a segmentation model based on graph convolutional network (GCN). For the single object segmentation task on cityscapes dataset, the GCN-based segmentation contour is used to generate a contour of a single object, then our contour renderer focuses on the pixels around the contour and predicts the category at high resolution. By rendering the contour result, our method reaches 72.41% mean intersection over union (IoU) and surpasses baseline Polygon-GCN by 1.22%.
BibTeX
@inproceedings{chen2020contourrend,
  author = {Junwen Chen and Yi Lu and Yaran Chen and Dongbin Zhao and Zhonghua Pang},
  title = {{ContourRend: A Segmentation Method for Improving Contours by Rendering}},
  booktitle = {ISNN},
  year = {2020},
  url = {https://openreview.net/forum?id=tp9_3oDHLa},
}
Figure