Abstract
Over the past decade, significant advances have been made in the field of image search for e-commerce applications. Traditional image-to-image (I2I) retrieval models, which focus solely on visual details such as texture, tend to overlook useful semantic information. Consequently, retrieved products might possess similar visual features but fail to fulfill the user's search intent. Moreover, using I2I models for products with multiple images results in significant storage overhead and complex mapping implementations. This paper reports the design and deployment of the Multi-modal Item Embedding Model (MIEM) to address these limitations. MIEM utilizes both textual information and multiple product images to construct meaningful semantic features. Since its deployment at Shopee in March 2023, it has achieved a 9.90% increase in clicks per user and a 4.23% boost in orders per user.
The Semantic Gap in Visual Search
Traditional e-commerce image search relies heavily on visual similarity. While effective for matching patterns, these systems often suffer from a lack of semantic grounding. For instance, a user searching for dental floss might be presented with battery chargers because both items are packaged in similar white-and-blue plastic containers. This visual ambiguity leads to irrelevant search results that frustrate users.
Furthermore, modern e-commerce listings typically include multiple images (front, back, side views). Storing separate embeddings for every image in a product's gallery leads to a massive search index, increasing infrastructure costs and requiring complex post-retrieval logic to prevent the same product from appearing multiple times in the results.
Methodology: The MIEM Architecture
To bridge the semantic gap and improve efficiency, the authors proposed MIEM, a dual-tower architecture designed for high-recall retrieval.
Dual-Tower Design
- Query Tower: Processes the user's uploaded image using a Swin Transformer backbone to extract visual features.
- Item Tower: Processes the product's metadata. It takes the product title (via Multilingual BERT) and up to four representative product images (via a shared Swin Transformer backbone).
Merge Attention and Fusion
Rather than using standard cross-attention, which can be computationally expensive for recall, MIEM employs a 6-layer Merge Attention Transformer. This module concatenates text and image tokens into a single sequence, allowing the model to learn complex inter-modal relationships. This "merging" approach is specifically optimized to preserve the unique information of both modalities for the retrieval phase.
Training Strategy
Training is conducted in three distinct stages to ensure robust feature extraction:
- Alignment: Initial alignment of image and text features using InfoNCE loss.
- Fusion Training: Training the fusion module using a single image and text pair. A novel Modality Balance Loss is introduced here to prevent the model from becoming over-reliant on one modality (e.g., ignoring text because images provide a stronger initial signal).
- Multi-Image Finetuning: The final stage incorporates multiple images ($K=4$) to capture a comprehensive, multi-angle view of the product.
System Implementation and Efficiency
Beyond accuracy, MIEM provides significant engineering advantages. By fusing multiple images and text into a single embedding per product, Shopee was able to reduce its Elasticsearch index size by 61.17%. This consolidation simplifies the search pipeline by removing the need for deduplication logic, as each product now occupies only one slot in the retrieval index.
Key Results
MIEM was evaluated through both offline metrics and online A/B testing on Shopee’s Indonesia and Brazil marketplaces:
- Business Impact: The model drove a 9.90% increase in clicks per user and a 4.23% increase in orders per user in the Indonesia market.
- Retrieval Accuracy: In offline benchmarks against a gallery of 3 million products, the MIEM+I2I hybrid achieved a Recall@1 of 0.872, compared to 0.820 for the standalone I2I model.
- Relevance: The "Top 4 Irrelevant Item Rate"—a measure of how many of the first four results are completely unrelated to the query—dropped from 13.58% to 10.71%.
Conclusion
The Multi-modal Item Embedding Model (MIEM) represents a significant step forward for e-commerce search by successfully integrating visual and textual signals. By moving beyond simple visual matching to semantic understanding, the system provides more relevant results to users while simultaneously reducing the computational and storage overhead of the search infrastructure.
Summary updated