Inferring the Importance of Product Appearance: A Step Towards the Screenless Revolution

Jiayu ZhouarXiv 2019 · arXiv

TL;DR

This paper tackles the challenge of determining which products are suitable for "screenless shopping" (e.g., via voice assistants) by inferring the importance of product appearance in consumer decisions. They propose a novel multi-view semi-supervised learning framework that leverages intrinsic features, images, and user comments to classify items as "strong appearance-related" (SA) or "weak appearance-related" (WA). This work is crucial for the future of e-commerce as it enables personalized shopping experiences and optimizes product design strategies.

What Problem Does This Paper Address?

The rapid growth of the Internet of Things (IoT) and smart devices is paving the way for "screenless retail," where interactions and purchases are made without traditional screens. While this offers new opportunities, not all products are equally suited for this paradigm. For items like clothing, dolls, or shoes, visual appearance is a critical factor in consumer decision-making. Without the ability to see the product, consumers may be hesitant to purchase. This paper addresses the fundamental problem of identifying which products are "strong appearance-related" (SA) and thus less suitable for screenless shopping, versus "weak appearance-related" (WA) items that are more amenable to this new shopping modality. This distinction is vital for both e-commerce platforms looking to optimize their offerings and for manufacturers aiming to prioritize design efforts.

Methods

The authors propose a novel multi-view semi-supervised learning framework to tackle the SA/WA classification problem. The core idea is to leverage information from three distinct "views" of a product:

  • Intrinsic Feature View: This includes higher-level product categories and basic attributes like size.
  • Comment Feature View: This captures user opinions and sentiments from product reviews, processed using Word2Vec embeddings and a keyword matching vector related to appearance.
  • Image Feature View: This uses deep learning (ResNet-50) to extract embeddings from product images.

The framework operates in two phases:

  1. Phase 1 (Supervised Training): A standard softmax loss is used for labeled data, combined with a view alignment loss to encourage the comment and image views to map into a similar feature space. This view alignment loss is particularly useful for handling "cold-start" scenarios where one view might be missing.
  2. Phase 2 (Semi-supervised Training): The model trained in Phase 1 is used to generate pseudo-labels for unlabeled data. A pseudo-labeled triplet loss is then introduced to further refine the model by enforcing constraints on the relationships between anchor, positive, and negative samples within the pseudo-labeled set. This loss is designed to be robust to potential errors in pseudo-labels.

The overall training process is carefully orchestrated to integrate these losses effectively.

Datasets

The study utilizes a large-scale, real-world transaction dataset collected from JD.com, a major Chinese online retailer. This dataset comprises:

  • 2,707 fine-grained categories of items.
  • One image per item.
  • Over 1 million user comments.

The categories were annotated via crowdsourcing, with 70% of these categories used for training and the remaining 30% for evaluation. A subset of 30% of the training data was initially designated as the labeled dataset for the semi-supervised learning process.

Key Insights and Ideas

  • Appearance Matters, But Not for Everything: The core insight is that product appearance is a significant decision factor for some items but not others, and this distinction can be learned from data. This directly informs the feasibility of screenless shopping for different product categories.
  • Multi-View Fusion is Key: Combining intrinsic features, user comments, and images provides a more comprehensive understanding of a product's appeal than any single view alone. User comments, in particular, offer valuable subjective insights into appearance importance.
  • Semi-Supervised Learning for Data Scarcity: The problem inherently faces a "limited data" challenge because real-world screenless shopping data is scarce. The proposed semi-supervised approach effectively leverages abundant unlabeled data to overcome this.
  • View Alignment for Robustness: The view alignment loss is a critical component, not only for fusing information but also for handling missing views (cold-start problem), making the model more resilient.
  • Pseudo-Labels Need Careful Handling: The introduction of a pseudo-labeled triplet loss highlights the need for robust methods when dealing with potentially noisy pseudo-labels generated from unlabeled data.

Key Results

  • The proposed method achieved a classification accuracy of 84.916%, significantly outperforming baseline methods like MLAN_C (72.920%), MLAN_SC (75.997%), and SWA(LSM) (72.435%).
  • The intrinsic feature view was found to contain the most useful information among the single views.
  • The model demonstrated strong performance in cold-start scenarios (where comment data was missing), achieving 82.389% accuracy, outperforming SWA (Joint) (74.319%) and SWA (LSM) (61.936%).
  • The well-designed training process (integrating losses sequentially) was crucial for achieving the best performance, as a naive combination of losses led to a significant drop in accuracy.
  • A case study showed the model correctly identifying items like shirts and wallets as SA categories with high confidence, while also demonstrating nuanced predictions for items like "second-hand" laptops, where the "second-hand" tag reduced the perceived importance of appearance.

Limitations and Future Work

  • User-Independence Assumption: The current study assumes that the importance of product appearance is user-independent. However, the authors acknowledge that individual preferences can vary (e.g., some might prioritize functionality over appearance for an umbrella).
  • Personalized Opinions: Future work will focus on learning personalized opinions about screenless shopping, considering that different users might have different subjective criteria for accepting screenless purchases.

Summary updated

Abstract

Nowadays, almost all the online orders were placed through screened devices such as mobile phones, tablets, and computers. With the rapid development of the Internet of Things (IoT) and smart appliances, more and more screenless smart devices, e.g., smart speaker and smart refrigerator, appear in our daily lives. They open up new means of interaction and may provide an excellent opportunity to reach new customers and increase sales. However, not all the items are suitable for screenless shopping, since some items' appearance play an important role in consumer decision making. Typical examples include clothes, dolls, bags, and shoes. In this paper, we aim to infer the significance of every item's appearance in consumer decision making and identify the group of items that are suitable for screenless shopping. Specifically, we formulate the problem as a classification task that predicts if an item's appearance has a significant impact on people's purchase behavior. To solve this problem, we extract features from three different views, namely items' intrinsic properties, items' images, and users' comments, and collect a set of necessary labels via crowdsourcing. We then propose an iterative semi-supervised learning framework with three carefully designed loss functions. We conduct extensive experiments on a real-world transaction dataset collected from the online retail giant JD.com. Experimental results verify the effectiveness of the proposed method.
BibTeX
@misc{zhou2019inferring,
  author = {Jiayu Zhou},
  title = {{Inferring the Importance of Product Appearance: A Step Towards the Screenless Revolution}},
  year = {2019},
  eprint = {1905.03698},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.03698},
}
Figure