Abstract
Over the past decade, significant advances have been made in the field of image search for e-commerce applications. Traditional image-to-image retrieval models, which focus solely on image details such as texture, tend to overlook useful semantic information contained within the images. As a result, the retrieved products might possess similar image details, but fail to fulfil the user's search goals. Moreover, the use of image-to-image retrieval models for products containing multiple images results in significant online product feature storage overhead and complex mapping implementations. In this paper, we report the design and deployment of the proposed Multi-modal Item Embedding Model (MIEM) to address these limitations. It is capable of utilizing both textual information and multiple images about a product to construct meaningful product features. By leveraging semantic information from images, MIEM effectively supplements the image search process, improving the overall accuracy of retrieval results. MIEM has become an integral part of the Shopee image search platform. Since its deployment in March 2023, it has achieved a remarkable 9.90% increase in terms of clicks per user and a 4.23% boost in terms of orders per user for the image search feature on the Shopee e-commerce platform.
Introduction
The evolution of e-commerce has made visual search a critical tool for users. However, traditional "Image-to-Image" (I2I) retrieval systems often struggle with semantic ambiguity. While these models excel at matching colors and textures, they lack the context necessary to distinguish between visually similar but functionally different products. For instance, a user searching for dental floss might be presented with battery chargers because both products share similar white plastic packaging.
Furthermore, modern product listings typically include multiple images. Storing and managing separate embeddings for every image per product creates massive storage overhead and complicates deduplication logic. To address these challenges, researchers at Shopee and Nanyang Technological University developed the Multi-modal Item Embedding Model (MIEM). This system fuses textual metadata (titles) with multiple product images into a single, semantically rich "item embedding" that is both more accurate and efficient.
Methodology: The MIEM Architecture
MIEM utilizes a dual-tower architecture designed to bridge the gap between visual features and semantic meaning:
- Dual-Tower Design: The "Query Tower" processes the user's uploaded image, while the "Item Tower" processes the product's title and its collection of images.
- Encoders: Visual features are extracted using a Swin Transformer, while product titles are processed via a multilingual BERT model.
- Merge Attention: To fuse these modalities, the authors implemented a 6-layer Merge Attention Transformer. Unlike standard cross-attention often reserved for final ranking, Merge Attention is used during the high-volume "recall" phase to ensure that both textual and visual information are preserved in the final embedding.
Training Strategy
To ensure robust performance, the model was trained using several advanced techniques:
- AM-InfoNCE Loss: An angular margin was added to the standard contrastive loss, creating sharper decision boundaries between similar-looking but distinct products.
- Modality Balance Loss: This novel function ensures the model learns independently from both text and images, preventing it from over-relying on a single modality (e.g., ignoring the text if the image is high-quality).
- Cross Batch Memory (XBM): This allows the model to compare items against a larger pool of "hard negatives" from previous batches, significantly improving training efficiency.
Experimental Setup and Datasets
The model was trained on real-world Shopee user click logs, consisting of triplets: <query image, clicked item title, clicked item image list>. The dataset revealed that while 45% of pairs were identical items, 50% were "similar" items, providing a rich ground for learning semantic associations.
Evaluation was conducted on a test set of 3 million products from the Shopee Indonesia market. Additionally, 9,000 query images were manually annotated by experts to establish ground-truth metrics for "Same," "Similar," or "Dissimilar" status.
Results and Impact
The deployment of MIEM on the Shopee platform in March 2023 led to significant improvements in both search relevance and business metrics.
Business Performance
In A/B testing conducted in Indonesia, MIEM drove a 9.90% increase in clicks per user and a 4.23% increase in orders per user. Similar success was observed in the Brazil market, with a 6.00% increase in clicks and a 4.95% increase in orders.
Retrieval Accuracy
Offline tests demonstrated that the hybrid MIEM+I2I approach achieved a Recall@1 of 0.872, outperforming standalone image models. Crucially, the "Top 4 Irrelevant Item Rate" dropped from 13.58% to 10.71%, indicating that users are much less likely to encounter "junk" results.
Efficiency Gains
By representing a product with a single multi-modal embedding instead of multiple individual image embeddings, MIEM reduced storage requirements by approximately 61% compared to the previous I2I system.
Conclusion and Future Work
MIEM demonstrates that integrating textual metadata as a "semantic anchor" significantly improves the precision of visual search. By moving beyond simple visual similarity, the model effectively resolves category confusion and reduces storage overhead.
Future work will focus on handling high-variance products (e.g., phone cases with hundreds of patterns) where a single embedding may still be insufficient. The authors also plan to explore Federated Learning to enhance user data privacy while maintaining the model's high performance.
Summary updated