Is a Picture Worth a Thousand Words? A Deep Multi-Modal Architecture for Product Classification in E-Commerce

Tom Zahavy, Abhinandan Krishnan, Alessandro Magnani, Shie Mannor

Abstract

Accurately categorizing millions of products into a complex taxonomy is a critical task for e-commerce platforms. While product titles provide rich information, they are often noisy or incomplete. Images offer a complementary source of data but are frequently insufficient on their own. This paper introduces a multi-modal deep learning architecture that utilizes a decision-level fusion strategy. By training a deep policy network to act as a "referee" between specialized text and image experts, the system learns to dynamically select the most reliable prediction. Evaluated on a massive dataset of 1.2 million products from Walmart.com, the proposed model achieves a significant accuracy improvement over single-modality baselines, demonstrating the effectiveness of learned fusion in large-scale industrial applications.

Introduction: The Scale of the Problem

In modern e-commerce, products must be precisely categorized into "shelves" to ensure they appear in relevant search results and recommendations. With millions of new items uploaded daily by various vendors, manual classification is no longer feasible.

The product classification pipeline involves processing massive daily uploads from various vendors into a structured taxonomy.

While machine learning models can automate this process, they face "messy" real-world data. Product titles can be vague, misspelled, or overly technical, while images might be low-quality or fail to distinguish between functionally different but visually similar items. The core challenge is effectively combining these two modalities, especially when one (text) is generally more reliable than the other (images).

Methodology: Decision-Level Fusion

The researchers from Walmart Labs and The Technion proposed a decision-level fusion architecture. Unlike feature-level fusion, which merges data early in the pipeline, this approach allows each modality to be processed by a specialized "expert" before a final decision is made.

  1. Text Expert: A CNN architecture optimized for capturing local patterns and keywords in product titles.
  2. Image Expert: A VGG16 model pre-trained on ImageNet and fine-tuned on the specific product categories.
  3. The Policy Network: This is the central innovation. It is a separate neural network that takes the output probabilities from both experts and learns a policy to decide which prediction to trust for a given product.

This "referee" approach prevents the weaker image model from introducing noise into the stronger text model's predictions unless the image model shows high confidence in a scenario where the text is ambiguous.

Key Insights and Results

The study utilized a dataset of 1.2 million products across nearly 3,000 categories. The results highlighted a clear hierarchy in modality strength: the text-only model achieved 70.1% accuracy, while the image-only model reached 56.7%.

Examples illustrating the varying informativeness of titles and images across different product categories.

The multi-modal policy network successfully bridged this gap, reaching 71.8% accuracy. This improvement stems from the model's ability to identify the ~8% of cases where the image provides the correct classification while the text fails. For instance, a product titled "WEAR BOOTS" might be a baby onesie with that text printed on it; a text-only model might misclassify it as footwear, whereas the image expert correctly identifies the garment type.

Conclusion

The research demonstrates that while "text is king" in e-commerce classification, images serve as a vital safety net. By moving beyond simple averaging or early concatenation to a learned decision-level policy, the system can effectively navigate the noise inherent in large-scale product data. Future work will explore deeper integration of raw features into the policy network to further close the gap toward the theoretical "optimal policy" of 77.7% accuracy.

Summary updated

Abstract

Classifying products precisely and efficiently is a major challenge in modern e-commerce. The high traffic of new products uploaded daily and the dynamic nature of the categories raise the need for machine learning models that can reduce the cost and time of human editors. In this paper, we propose a decision level fusion approach for multi-modal product classification based on text and image neural network classifiers. We train input specific state-of-the-art deep neural networks for each input source, show the potential of forging them together into a multi-modal architecture and train a novel policy network that learns to choose between them. Finally, we demonstrate that our multi-modal network improves classification accuracy over both networks on a real-world large-scale product classification dataset that we collected from Walmart.com. While we focus on image-text fusion that characterizes e-commerce businesses, our algorithms can be easily applied to other modalities such as audio, video, physical sensors, etc.
BibTeX
@inproceedings{zahavy2018is,
  author = {Tom Zahavy and Abhinandan Krishnan and Alessandro Magnani and Shie Mannor},
  title = {{Is a Picture Worth a Thousand Words? A Deep Multi-Modal Architecture for Product Classification in E-Commerce}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11419},
}
Figure