Towards Predicting the Likeability of Fashion Images

Jinghua Wang, Abrar Abdul Nabi, Gang Wang, Chengde Wan, Tian-Tsong NgarXiv 2015 · OpenReview

TL;DR

This paper tackles the challenge of automatically ranking fashion images based on their predicted likeability, crucial for online shopping platforms. They introduce a novel approach using both semantic and data-driven attributes, learned via multi-task CNNs and an unsupervised method, respectively. These attributes are then fed into a Sum-Product Network (SPN) to model complex attribute correlations and predict image ranking, outperforming traditional methods.

What Problem Does This Paper Address?

The explosive growth of online shopping, particularly in the fashion sector, has created a demand for intelligent systems that can automatically curate and present visually appealing products. Currently, platforms often rely on human experts to select featured images, a process that is subjective, time-consuming, and difficult to scale. The core problem this paper addresses is how to automatically predict which fashion images are most likely to be liked by a large number of users, thereby improving user engagement and product discovery on e-commerce and social media platforms. This involves understanding the visual cues that contribute to an image's appeal, which can be subtle and complex.

Methods

The authors propose a multi-stage approach:

  1. Attribute Learning:
    • Semantic Attributes: They employ a multi-task Convolutional Neural Network (CNN) to learn semantic attributes (e.g., "color is red," "floral pattern"). This approach leverages shared layers to transfer visual knowledge across different attribute categories, improving performance, especially with limited training data for individual attributes.
  2. Data-Driven Attributes: To capture visual patterns that lack explicit names (e.g., specific textures, subtle design elements), they develop an unsupervised method. This involves iteratively extracting features from image patches using a CNN, clustering these features to identify visual patterns, and then fine-tuning the CNN to better represent these discovered patterns. Each cluster is treated as a data-driven attribute.
  3. Image Representation: Fashion images are represented as attribute vectors, combining the activations of both semantic and data-driven attributes.
  4. Ranking with Sum-Product Networks (SPNs): To model the high-order correlations between attributes (where combinations, not just individual attributes, influence likeability), they propose learning a ranking SPN. SPNs are deep probabilistic models capable of capturing complex dependencies. The SPN is trained to output a higher value for images predicted to be more liked. A novel learning method is introduced to update SPN parameters based on pairs of images, aiming to increase the difference in output values for pairs with significantly different likeability scores and decrease it for pairs with similar scores.

Datasets

The authors introduce two new datasets for fashion image likeability prediction:

  • Pinterest Dataset: Collected from Pinterest.com, focusing on "Women's fashion."
  • Polyvore Dataset: Collected from Polyvore.com, a social commerce website.

Both datasets consist of dress images and their associated likeability scores (number of likes). They collected 6,673 images from Pinterest and 69,256 from Polyvore. Images were segmented to isolate dresses.

Key Insights and Ideas

  • Data-Driven Attributes are Crucial: The research strongly suggests that "data-driven" attributes, which capture subtle visual patterns without explicit names, are more discriminative for predicting fashion image likeability than traditional semantic attributes. This is likely because they can capture nuanced aesthetic qualities.
  • High-Order Attribute Correlations Matter: The success of the SPN highlights that fashion appeal isn't just about individual features but how they combine. An SPN's ability to model these complex, non-linear interactions is key to its effectiveness.
  • Unsupervised Discovery of Visual Patterns: The unsupervised method for discovering data-driven attributes is a significant contribution, enabling the extraction of useful visual cues without manual annotation, which is a bottleneck in many computer vision tasks.
  • Multi-Task Learning for Attribute Robustness: Using multi-task CNNs for semantic attribute learning effectively shares knowledge, leading to more robust attribute classifiers, especially when training data is scarce.

Key Results

  • Improved Ranking Accuracy: The proposed SPN-based method, utilizing both semantic and data-driven attributes, significantly outperforms traditional ranking methods like RankSVM and structured RankSVM, as well as SPNs using only semantic or data-driven attributes. For instance, on the Polyvore dataset, their SPN achieves an accuracy of 82.1% with 192 data-driven attributes, compared to RankSVM with deep features at 72.2%.
  • Effectiveness of Data-Driven Attributes: The paper demonstrates that using data-driven attributes alone can achieve competitive or superior results compared to semantic attributes, and combining them further boosts performance.
  • Attribute Combinations Impact Likeability: Qualitative results show that certain combinations of data-driven attributes (e.g., specific patterns on a dress) can significantly enhance likeability, while others can decrease it, underscoring the importance of modeling these interactions.

Limitations and Future Work

  • Parameter Sensitivity: The performance of the ranking system is sensitive to the parameter $\theta$, which defines the minimum difference in likes to consider a pair of images for ranking. A smaller $\theta$ leads to a larger testing set and a more difficult ranking task, but also reduces the reliability of the ranking order itself.
  • Computational Cost: While SPNs offer faster inference than some complex models, learning their structure and parameters can still be computationally intensive.
  • Future Work: The authors suggest exploring more sophisticated SPN structures and learning algorithms. They also mention the potential for discovering attractive and unattractive attribute sets, which could lead to more interpretable insights into fashion appeal. Further investigation into how fashion trends evolve over time and how to incorporate this into the model is also a promising direction.

Summary updated

Abstract

In this paper, we propose a method for ranking fashion images to find the ones which might be liked by more people. We collect two new datasets from image sharing websites (Pinterest and Polyvore). We represent fashion images based on attributes: semantic attributes and data-driven attributes. To learn semantic attributes from limited training data, we use an algorithm on multi-task convolutional neural networks to share visual knowledge among different semantic attribute categories. To discover data-driven attributes unsupervisedly, we propose an algorithm to simultaneously discover visual clusters and learn fashion-specific feature representations. Given attributes as representations, we propose to learn a ranking SPN (sum product networks) to rank pairs of fashion images. The proposed ranking SPN can capture the high-order correlations of the attributes. We show the effectiveness of our method on our two newly collected datasets.
BibTeX
@misc{wang2015towards,
  author = {Jinghua Wang and Abrar Abdul Nabi and Gang Wang and Chengde Wan and Tian-Tsong Ng},
  title = {{Towards Predicting the Likeability of Fashion Images}},
  year = {2015},
  eprint = {1511.05296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.05296},
}
Figure