Abstract
Accurately categorizing millions of products into a complex taxonomy is a critical task for e-commerce platforms. While product titles provide rich information, they are often noisy or incomplete. Images offer a complementary source of data but are frequently insufficient on their own. This paper introduces a multi-modal deep learning architecture that utilizes a decision-level fusion strategy. By training a deep policy network to act as a "referee" between specialized text and image experts, the system learns to dynamically select the most reliable prediction. Evaluated on a massive dataset of 1.2 million products from Walmart.com, the proposed model achieves a significant accuracy improvement over single-modality baselines, demonstrating the effectiveness of learned fusion in large-scale industrial applications.
Introduction: The Scale of the Problem
In modern e-commerce, products must be precisely categorized into "shelves" to ensure they appear in relevant search results and recommendations. With millions of new items uploaded daily by various vendors, manual classification is no longer feasible.
While machine learning models can automate this process, they face "messy" real-world data. Product titles can be vague, misspelled, or overly technical, while images might be low-quality or fail to distinguish between functionally different but visually similar items. The core challenge is effectively combining these two modalities, especially when one (text) is generally more reliable than the other (images).
Methodology: Decision-Level Fusion
The researchers from Walmart Labs and The Technion proposed a decision-level fusion architecture. Unlike feature-level fusion, which merges data early in the pipeline, this approach allows each modality to be processed by a specialized "expert" before a final decision is made.
- Text Expert: A CNN architecture optimized for capturing local patterns and keywords in product titles.
- Image Expert: A VGG16 model pre-trained on ImageNet and fine-tuned on the specific product categories.
- The Policy Network: This is the central innovation. It is a separate neural network that takes the output probabilities from both experts and learns a policy to decide which prediction to trust for a given product.
This "referee" approach prevents the weaker image model from introducing noise into the stronger text model's predictions unless the image model shows high confidence in a scenario where the text is ambiguous.
Key Insights and Results
The study utilized a dataset of 1.2 million products across nearly 3,000 categories. The results highlighted a clear hierarchy in modality strength: the text-only model achieved 70.1% accuracy, while the image-only model reached 56.7%.
The multi-modal policy network successfully bridged this gap, reaching 71.8% accuracy. This improvement stems from the model's ability to identify the ~8% of cases where the image provides the correct classification while the text fails. For instance, a product titled "WEAR BOOTS" might be a baby onesie with that text printed on it; a text-only model might misclassify it as footwear, whereas the image expert correctly identifies the garment type.
Conclusion
The research demonstrates that while "text is king" in e-commerce classification, images serve as a vital safety net. By moving beyond simple averaging or early concatenation to a learned decision-level policy, the system can effectively navigate the noise inherent in large-scale product data. Future work will explore deeper integration of raw features into the policy network to further close the gap toward the theoretical "optimal policy" of 77.7% accuracy.
Summary updated