Abstract
Existing deep learning based interactive segmentation methods have achieved remarkable performance with only a few user clicks. However, state-of-the-art methods often struggle with objects containing elongated thin structures, such as bicycle spokes or insect legs. We identify two primary causes: a lack of appropriate training data and extreme class imbalance between thin and non-thin regions. To address these, we introduce ThinObject-5K, a large-scale dataset specifically for thin objects, and TOS-Net, a novel three-stream network. TOS-Net combines a high-resolution edge stream for fine details with a fixed-resolution context stream for semantic understanding. Our approach surpasses baselines by approximately 30% in $IoU_{thin}$ using only four clicks.
The "Thinness Problem" in Interactive Segmentation
Interactive image segmentation typically relies on user-provided "extreme clicks" to extract objects. While models like DEXTR perform exceptionally well on standard benchmarks, they frequently produce "blobby" masks when faced with thin structures. For instance, in a bicycle image, the model might capture the frame but fail to segment the spokes.
This failure stems from two main issues:
- Coarse Training Data: Popular datasets like PASCAL VOC and MS COCO often ignore fine details in their annotations to save time, effectively teaching models to overlook thin parts.
- Class Imbalance: Standard loss functions are dominated by the large number of pixels in an object's body, making the loss from missing a few thin-structure pixels statistically negligible.
TOS-Net: A Three-Stream Architecture
The proposed Thin Object Selection Network (TOS-Net) employs a divide-and-conquer strategy through three specialized streams:
- Context Stream: Operates at a fixed, lower resolution to capture the overall semantic context and provide a rough initial mask.
- High-Resolution Edge Stream: Processes the image at its native resolution to preserve fine-grained details. It uses image gradients and semantic features from the context stream to delineate sharp boundaries.
- Fusion Stream: Amalgamates the outputs using RoIAlign to ensure spatial alignment, resulting in a final mask with sharp predictions around thin parts.
Additionally, TOS-Net uses an Adaptive Relaxation scheme, scaling the search area based on the object's size rather than using a fixed padding, which ensures sufficient context for objects of all scales.
The ThinObject-5K Dataset
To overcome the data quality bottleneck, the authors created ThinObject-5K, featuring 5,743 unique objects with elongated structures (e.g., rackets, cables, nets). These annotations were meticulously refined in Photoshop and composited onto diverse backgrounds. Training on this high-fidelity data alone provides a significant boost in performance for thin object selection.
Experimental Results
TOS-Net demonstrates substantial improvements, particularly in the $IoU_{thin}$ metric, which specifically evaluates accuracy around thin regions. On the HRSOD dataset, TOS-Net achieved an $IoU_{thin}$ of 65.1%, significantly outperforming traditional interactive segmentation models.
Key findings include:
- Data Impact: Simply training existing architectures on ThinObject-5K improves thin-object IoU by nearly 30%.
- Efficiency: The model processes 2K images in approximately 0.45s on a Titan Xp, making it suitable for real-time interactive use.
Conclusion and Future Work
TOS-Net and the ThinObject-5K dataset represent a significant step forward in high-fidelity interactive segmentation. While the model excels at thin structures, it can occasionally be over-eager, segmenting background thin objects (like fences) if they overlap with the target. Future research will focus on resolving these ambiguities and improving performance on semi-transparent objects.
Summary updated