Abstract
Nowadays, live-stream and short video shopping in E-commerce have grown exponentially. However, the sellers are required to manually match images of the selling products to the timestamp of exhibition in the untrimmed video, resulting in a complicated process. To solve the problem, we present an innovative demonstration of multi-modal retrieval system called "Fashion Focus", which enables to exactly localize the product images in the online video as the focuses. Different modality contributes to the community localization, including visual content, linguistic features and interaction context are jointly investigated via presented multi-modal learning. Our system employs two procedures for analysis, including video content structuring and multi-modal retrieval, to automatically achieve accurate video-to-shop matching. Fashion Focus presents a unified framework that can orientate the consumers towards relevant product exhibitions during watching videos and help the sellers to effectively deliver the products over search and recommendation.
Introduction
The explosion of live-stream shopping has introduced a significant metadata bottleneck for e-commerce platforms. Currently, sellers must manually synchronize product listings with specific timestamps in hours-long videos to ensure "buy" links appear correctly. This manual process is tedious and prone to errors. From a technical standpoint, this "video-to-shop" retrieval task is complicated by dynamic environments, a long-tail distribution of millions of similar fashion items, and the need for real-time processing at scale.
The Fashion Focus Framework
To address these challenges, the "Fashion Focus" system utilizes a two-stage framework designed to automate the matching of video segments to product images.
1. Video Content Structuring
Rather than analyzing isolated frames, the system generates tracklet proposals. Using a one-stage video detector with a DLA-34 backbone, the system jointly performs object detection and appearance embedding. This allows it to follow a specific item across multiple frames, creating a "tracklet" that provides a more robust visual representation than a single image. A hierarchical classification approach is employed to mitigate category ambiguity in the long-tail distribution.
2. Multi-modal Retrieval and Fusion
The core of the system is the fusion of disparate data streams to accurately "triangulate" products. The system integrates:
- Visual Features: Extracted directly from the tracklets.
- Audio Data: Speech is converted to text via Automatic Speech Recognition (ASR).
- Linguistic/Contextual Data: User comments (e.g., "Can you show item No. 9?") and video headlines are mined for cues.
These modalities are fused using the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) model. This maps visual and linguistic features into a single compact embedding space, optimized using a triplet constraint to ensure matching video segments and product images are mathematically close.
Experimental Evaluation
The system was evaluated on a large-scale dataset from Alibaba (Taobao), consisting of 1,000 live-stream videos (5–7 hours each) across categories like clothing, shoes, and beauty products. The primary metric was the recall of identical matched products.
The results demonstrate that the multi-modal tracklet approach (Mlt_Trk) significantly outperforms baseline visual-only methods:
| Method | Clothing | Shoe | Bag | Mean Recall |
|---|---|---|---|---|
| Vis_Img (Single Image) | 16.4% | 15.1% | 13.7% | 12.5% |
| Vis_Trk (Visual Tracklet) | 19.3% | 17.1% | 16.4% | 14.7% |
| Mlt_Trk (Multi-modal) | 23.6% | 19.2% | 18.3% | 16.8% |
The inclusion of temporal tracking (Vis_Trk) improves performance over static images, but the multi-modal integration provides the most substantial boost, particularly in the clothing category where visual ambiguity is highest.
Conclusion
Fashion Focus demonstrates that "noisy" live-stream data, such as audio explanations and user comments, can serve as valuable "weak annotations" to improve retrieval accuracy. While a mean recall of 16.8% highlights the ongoing difficulty of automated video-to-shop matching, the system provides a scalable foundation for reducing manual labor for sellers and enhancing the consumer shopping experience. Future work will focus on improving real-time efficiency and addressing the "vanishing-classification" problem for rare items.
Summary updated