DeepMark++: Real-Time Clothing Detection at the Edge

Alexey Sidnev, Alexander Krapivin, Alexey Trushkov, Ekaterina Krasikova, Maxim Kazakov, Mikhail ViryasovWACV 2021

TL;DR

This paper introduces DeepMark++, a novel single-stage approach for real-time clothing detection and keypoint estimation on edge devices. By adapting the CenterNet architecture and introducing semantic keypoint grouping and several post-processing techniques, DeepMark++ achieves state-of-the-art accuracy on the DeepFashion2 dataset while running efficiently on mobile hardware.

What Problem Does This Paper Address?

Clothing recognition is a fundamental challenge in the fashion domain, with applications ranging from virtual try-on to automated fashion analysis. While existing methods can achieve good recognition accuracy, they often suffer from slow inference speeds and high computational demands, making them unsuitable for deployment on resource-constrained edge devices like smartphones. This paper aims to bridge this gap by developing a system that is both accurate and efficient enough for real-time performance on mobile platforms.

Methods

The core of DeepMark++ is built upon the CenterNet [26] architecture, a popular anchor-free object detection framework. The authors adapt CenterNet to simultaneously perform two tasks: object detection (predicting bounding boxes) and keypoint estimation (locating specific landmarks on clothing items).

Key novelties and improvements include:

  • Semantic Keypoint Grouping: The DeepFashion2 dataset has a large number of keypoints (294 in total) across 13 clothing classes. To manage this complexity, the authors propose grouping semantically similar keypoints across different categories. This reduces the number of output channels from 901 to 205, significantly decreasing training time and memory consumption without a substantial drop in accuracy.
  • Post-processing Techniques: Four post-processing techniques are introduced to enhance accuracy without compromising performance:
  • Center Rescoring: Recalculates detection confidence by combining the bounding box score with the average score of refined keypoints.
  • Heatmap Rescoring with Gaussian Kernel: Applies a Gaussian smoothing to heatmaps to improve the localization of peaks and their corresponding values.
  • Keypoint Location Refinement: A linear combination of refined and coarse keypoint positions to further correct locations.
  • Keypoint Heatmap Rescoring: Adds a penalty to keypoint scores based on their distance from a coarse keypoint position.
  • Lightweight Backbone: For edge deployment, a modified DLA-34 backbone (mDLA-34) is used, replacing Deformable Convolutional Networks (DCN) layers with standard convolutions to ensure compatibility with mobile inference frameworks.

Datasets

  • DeepFashion2 Dataset [4]: This is the primary dataset used for training and evaluation. It is noted for its large scale, diverse clothing classes (13), and detailed annotations of bounding boxes and keypoints (294 in total).
  • COCO Dataset: Used for evaluating the proposed post-processing techniques and for training the PoseFix refinement network.

Key Insights and Ideas

  • Semantic Grouping is Key for Efficiency: The insight that many clothing keypoints are semantically equivalent across different clothing types is crucial. Grouping these reduces the model's complexity and computational burden, making it feasible for edge devices.
  • Post-processing Matters for Accuracy: While the core network is important, the proposed post-processing techniques significantly boost accuracy, particularly for keypoint estimation, without requiring retraining of the entire model.
  • Trade-off Management: The paper effectively demonstrates the trade-off between speed and accuracy, showing how different configurations and techniques can be chosen to suit specific deployment scenarios (e.g., high accuracy vs. real-time mobile performance).
  • Mobile Deployment is Achievable: By carefully selecting and modifying network components (like replacing DCNs), the authors show that high-performance clothing analysis can be achieved on standard smartphones.

Key Results

  • State-of-the-Art Accuracy: The proposed approach achieves results comparable to state-of-the-art solutions on the DeepFashion2 dataset.
  • DeepFashion2 Landmark Estimation Challenge 2020: The model achieved second place in this challenge with a 0.582 mAP.
  • Real-time Mobile Performance: A light version of the model runs at 17 FPS on the Huawei P40 Pro smartphone with an acceptable accuracy of 0.445 mAP for keypoint estimation.
  • Efficiency Gains: Semantic keypoint grouping reduces training time and memory consumption by up to 26% and 28% respectively, and inference time and memory by up to 26% and 37%.
  • Post-processing Impact: Combined post-processing techniques improved mAPpt by 0.07, while only slightly impacting mAPbox.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the model achieves good performance, it can still produce overconfident incorrect predictions, a known issue with CenterNet.
  • Clothing detection on distant or blurred subjects can still be challenging.
  • Uncommon hand positions or partially visible body parts can lead to errors in keypoint determination.
  • Future Work:
    • The authors suggest exploring multi-inference strategies (e.g., fusing results from original and flipped images) to further boost accuracy, though this comes at the cost of increased processing time.
  • Further research into improving the robustness of clothing detection on challenging cases like distant or blurred subjects is implied.

Summary updated

BibTeX
@inproceedings{sidnev2021deepmark,
  author = {Alexey Sidnev and Alexander Krapivin and Alexey Trushkov and Ekaterina Krasikova and Maxim Kazakov and Mikhail Viryasov},
  title = {{DeepMark++: Real-Time Clothing Detection at the Edge}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure