Abstract
Nowadays, live-stream and short video shopping in E-commerce have grown exponentially. However, sellers are currently required to manually match product images to specific timestamps in untrimmed videos, a labor-intensive and complex process. To address this, we present "Fashion Focus," an innovative multi-modal retrieval system that enables exact localization of product images within online videos. By jointly investigating visual content, linguistic features, and interaction context through multi-modal learning, our system employs two key procedures—video content structuring and multi-modal retrieval—to achieve accurate video-to-shop matching automatically. Fashion Focus provides a unified framework that orientates consumers toward relevant product exhibitions and assists sellers in effectively delivering products via search and recommendation.
Introduction: The Video-to-Shop Challenge
In the rapidly evolving world of e-commerce, live-streaming has become a powerhouse for sales. However, a significant friction point remains: sellers must manually link products to specific timestamps in hours-long videos, and shoppers often struggle to find the exact item a host is currently holding.
The "video-to-shop" challenge is deceptively difficult. Unlike clean studio photos, products in live-streams are subject to varying viewpoints, motion blur, and background clutter. Manual tagging is labor-intensive; if a host showcases 100 items in a five-hour stream, finding item #42 manually is a poor experience for both seller and consumer. Previous visual search approaches often fail because they rely solely on pixels, ignoring the rich auxiliary data—spoken descriptions, headlines, and real-time user comments—that provide vital context for identification.
The Fashion Focus Framework
Fashion Focus is an end-to-end system designed to automate product identification by fusing visual tracklets with linguistic data. The framework consists of two primary stages:
- Video Content Structuring: Instead of treating every frame independently, the system uses a joint detection and tracking approach with a DLA-34 backbone. It generates "tracklet proposals"—sequences of frames following a specific object—to ensure temporal consistency and focus on the product being exhibited rather than background noise.
- Multi-modal Retrieval: This core innovation extracts features from three distinct streams:
- Visual Stream: CNN features derived from the tracklets.
- Linguistic Stream: Audio processed via Automatic Speech Recognition (ASR) to extract descriptive keywords.
- Interaction Stream: Real-time user comments (e.g., "Can you show item #9?") and headlines mined for contextual clues.
These modalities are fused into a compact embedding using LXMERT, a transformer-based model for learning vision-and-language representations. The system then performs retrieval by matching these multi-modal video embeddings against a database of static product images using triplet loss metric learning.
Experimental Evaluation
The researchers evaluated the system using a large-scale dataset from the Taobaolive platform, comprising 1,000 live-stream videos (5–7 hours each) across categories like clothing, shoes, bags, and beauty products. Ground truth was established using manually matched product-to-video timestamps.
The study compared three methods to measure the "Recall" of correctly localized commodities:
- Visual Image (Vis_Img): 12.5% mean recall.
- Visual Tracklet (Vis_Trk): 14.7% mean recall, demonstrating the value of temporal tracking.
- Multi-modal Tracklet (Mlt_Trk): 16.8% mean recall.
The multi-modal approach provided a significant boost across all categories, with the highest performance in Clothing (23.6% recall) and Shoes (19.2% recall), where visual and verbal descriptions tend to be the most descriptive.
Conclusion and Future Work
Fashion Focus demonstrates that context acts as a powerful "weak annotation." By leveraging audio and comments, the system narrows the search space more effectively than visual features alone. Using tracklets instead of static frames provides robustness against occlusion and blur, allowing the model to "see" the product from multiple angles as the host moves it.
While a 16.8% recall marks a significant improvement over baseline visual search, fully autonomous video-to-shop matching remains a difficult frontier. The authors note that "recognition at scale" still faces challenges with the long-tail distribution of products. Future work will focus on refining multi-modal fusion—perhaps by weighing the reliability of noisy comments—and further reducing latency for massive-scale, real-time streaming applications.
Summary updated