Deep Interactive Thin Object Selection

Jun Hao Liew, Scott Cohen, Brian L. Price, Long Mai, Jiashi FengWACV 2021 · OpenReview

Abstract

Existing deep learning based interactive segmentation methods have achieved remarkable performance with only a few user clicks. However, state-of-the-art methods often struggle with objects containing elongated thin structures, such as bicycle spokes or insect legs. We identify two primary causes: a lack of appropriate training data and extreme class imbalance between thin and non-thin regions. To address these, we introduce ThinObject-5K, a large-scale dataset specifically for thin objects, and TOS-Net, a novel three-stream network. TOS-Net combines a high-resolution edge stream for fine details with a fixed-resolution context stream for semantic understanding. Our approach surpasses baselines by approximately 30% in $IoU_{thin}$ using only four clicks.

The "Thinness Problem" in Interactive Segmentation

Interactive image segmentation typically relies on user-provided "extreme clicks" to extract objects. While models like DEXTR perform exceptionally well on standard benchmarks, they frequently produce "blobby" masks when faced with thin structures. For instance, in a bicycle image, the model might capture the frame but fail to segment the spokes.

Figure 1: Comparison of segmentation quality. Standard methods often produce "blobby" masks (left) that miss thin structures like spokes, whereas TOS-Net (right) captures fine details.

This failure stems from two main issues:

  1. Coarse Training Data: Popular datasets like PASCAL VOC and MS COCO often ignore fine details in their annotations to save time, effectively teaching models to overlook thin parts.
  2. Class Imbalance: Standard loss functions are dominated by the large number of pixels in an object's body, making the loss from missing a few thin-structure pixels statistically negligible.

TOS-Net: A Three-Stream Architecture

The proposed Thin Object Selection Network (TOS-Net) employs a divide-and-conquer strategy through three specialized streams:

  • Context Stream: Operates at a fixed, lower resolution to capture the overall semantic context and provide a rough initial mask.
  • High-Resolution Edge Stream: Processes the image at its native resolution to preserve fine-grained details. It uses image gradients and semantic features from the context stream to delineate sharp boundaries.
  • Fusion Stream: Amalgamates the outputs using RoIAlign to ensure spatial alignment, resulting in a final mask with sharp predictions around thin parts.

Additionally, TOS-Net uses an Adaptive Relaxation scheme, scaling the search area based on the object's size rather than using a fixed padding, which ensures sufficient context for objects of all scales.

The ThinObject-5K Dataset

To overcome the data quality bottleneck, the authors created ThinObject-5K, featuring 5,743 unique objects with elongated structures (e.g., rackets, cables, nets). These annotations were meticulously refined in Photoshop and composited onto diverse backgrounds. Training on this high-fidelity data alone provides a significant boost in performance for thin object selection.

Experimental Results

TOS-Net demonstrates substantial improvements, particularly in the $IoU_{thin}$ metric, which specifically evaluates accuracy around thin regions. On the HRSOD dataset, TOS-Net achieved an $IoU_{thin}$ of 65.1%, significantly outperforming traditional interactive segmentation models.

Figure 5: Quantitative comparison of IoU and $IoU_{thin}$ across different models on the HRSOD dataset. TOS-Net significantly outperforms baselines in capturing thin structures.

Key findings include:

  • Data Impact: Simply training existing architectures on ThinObject-5K improves thin-object IoU by nearly 30%.
  • Efficiency: The model processes 2K images in approximately 0.45s on a Titan Xp, making it suitable for real-time interactive use.

Conclusion and Future Work

TOS-Net and the ThinObject-5K dataset represent a significant step forward in high-fidelity interactive segmentation. While the model excels at thin structures, it can occasionally be over-eager, segmenting background thin objects (like fences) if they overlap with the target. Future research will focus on resolving these ambiguities and improving performance on semi-transparent objects.

Summary updated

Abstract

Existing deep learning based interactive segmentation methods have achieved remarkable performance with only a few user clicks, e.g. DEXTR [32] attaining 91.5% IoU on PASCAL VOC with only four extreme clicks. However, we observe even the state-of-the-art methods would often struggle in cases of objects to be segmented with elongated thin structures (e.g. bug legs and bicycle spokes). We investigate such failures, and find the critical reasons behind are two-fold: 1) lack of appropriate training dataset; and 2) extremely imbalanced distribution w.r.t. number of pixels belonging to thin and non-thin regions. Targeted at these challenges, we collect a large-scale dataset specifically for segmentation of thin elongated objects, named ThinObject-5K. Also, we present a novel integrative thin object segmentation network consisting of three streams. Among them, the high-resolution edge stream aims at preserving fine-grained details including elongated thin parts; the fixed-resolution context stream focuses on capturing semantic contexts. The two streams' outputs are then amalgamated in the fusion stream to complement each other for help producing a refined segmentation output with sharper predictions around thin parts. Extensive experimental results well demonstrate the effectiveness of our proposed solution on segmenting thin objects, surpassing the baseline by ~ 30% IoU <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">thin</sub> despite using only four clicks. Codes and dataset are available at https://github.com/liewjunhao/thin-object-selection.
BibTeX
@inproceedings{liew2021deep,
  author = {Jun Hao Liew and Scott Cohen and Brian L. Price and Long Mai and Jiashi Feng},
  title = {{Deep Interactive Thin Object Selection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=tbGDfjA5lTb},
}
Figure