Abstract
In this paper we present an extensive evaluation of instance segmentation in the context of images containing clothes. We propose a multi level evaluation that completes the classical overlapping criteria given by IoU. In particular, we quantify both the contour and color content accuracy of the the predicted segmentation masks. We demonstrate that the proposed evaluation framework is relevant to obtain meaningful insights on models performance through experiments conducted on five state of the art instance segmentation methods.
Introduction
Instance segmentation—the task of identifying and outlining every individual garment in an image—is a cornerstone for fashion applications such as virtual try-ons, automated clothes retrieval, and color harmony analysis. However, the industry-standard metric, mean Average Precision (mAP) based on Intersection over Union (IoU), is often too coarse for these nuanced tasks. IoU measures pixel-wise overlap but fails to distinguish between masks with jagged edges (contour errors) or those that include background elements (content errors). For high-fidelity applications like virtual try-ons, these quality differences are critical.
Proposed Multi-Level Evaluation Framework
To address the limitations of IoU, we propose a three-level evaluation framework designed to provide a more holistic view of model performance:
- Global Level (IoU): Measures traditional pixel-wise overlap to ensure the model identifies the correct general region.
- Contour Level (Boundary Jaccard - BJ): Evaluates the precision of the mask edges. By using a distance-based metric, BJ penalizes predictions that deviate from the true garment boundary, ensuring "clean" extractions necessary for realistic clothing overlays.
- Content Level (sEMD): Quantifies color distribution accuracy using the Earth Mover’s Distance (EMD) on 3D histograms in the Lab* color space. This metric detects "over-segmentation" (inclusion of background/skin) and "under-segmentation" (missing fabric) by comparing the color profile of the predicted mask to the ground truth.
Experimental Setup
The study utilizes a subset of the DeepFashion2 dataset, currently the largest and most comprehensive dataset for fashion-related tasks. The authors used a subset of 52,490 instances across 32,153 images for their evaluation, focusing specifically on the instance segmentation masks. We benchmarked five state-of-the-art architectures: Mask R-CNN, Mask Scoring R-CNN (MS R-CNN), Cascade Mask R-CNN, Hybrid Task Cascade (HTC), and Yolact.
Results and Discussion
Our multi-level analysis revealed several key insights into the performance of modern segmentation models:
- The "Blindness" of IoU: As illustrated in the figure above, masks with nearly identical IoU scores (around 0.61) can exhibit vastly different quality. For instance, $G_4$ achieves a high sEMD score (0.973) by focusing on the core fabric, while $G_3$ includes significant background noise, resulting in a much lower sEMD (0.644) despite a similar IoU.
- Performance Winner: Yolact, a real-time single-shot detector, surprisingly outperformed more complex multi-stage models across all metrics, achieving the highest $mIoU$ (0.854), $mBJ$ (0.737), and $msEMD$ (0.943).
- Efficiency: Yolact was significantly faster, processing images in 0.07 seconds, compared to 0.22 seconds for the runner-up, HTC. This makes it the most viable candidate for real-time fashion applications.
- Contour Refinement: MS R-CNN demonstrated improved contour accuracy ($mBJ$) over standard Mask R-CNN, validating the benefit of specifically supervising mask quality during the training process.
Conclusion and Future Work
The proposed multi-level framework provides a more granular and meaningful evaluation of instance segmentation quality than traditional overlap metrics. While effective, future work will address the computational cost of EMD and incorporate texture analysis to distinguish between different fabric patterns. We also plan to extend this evaluation to a broader range of single-shot architectures to further explore the trade-offs between speed and segmentation quality.
Summary updated