Fashion Focus: Multi-modal Retrieval System for Video Commodity Localization in E-commerce

Yanhao Zhang, Qiang Wang, Pan Pan, Yun Zheng, Cheng Da, Siyang Sun, Yinghui Xu

Abstract

Nowadays, live-stream and short video shopping in E-commerce have grown exponentially. However, sellers are currently required to manually match product images to specific timestamps in untrimmed videos, a labor-intensive and complex process. To address this, we present "Fashion Focus," an innovative multi-modal retrieval system that enables exact localization of product images within online videos. By jointly investigating visual content, linguistic features, and interaction context through multi-modal learning, our system employs two key procedures—video content structuring and multi-modal retrieval—to achieve accurate video-to-shop matching automatically. Fashion Focus provides a unified framework that orientates consumers toward relevant product exhibitions and assists sellers in effectively delivering products via search and recommendation.

Introduction: The Video-to-Shop Challenge

In the rapidly evolving world of e-commerce, live-streaming has become a powerhouse for sales. However, a significant friction point remains: sellers must manually link products to specific timestamps in hours-long videos, and shoppers often struggle to find the exact item a host is currently holding.

Application of Fashion Focus in a live-streaming scenario, showing real-time product links.

The "video-to-shop" challenge is deceptively difficult. Unlike clean studio photos, products in live-streams are subject to varying viewpoints, motion blur, and background clutter. Manual tagging is labor-intensive; if a host showcases 100 items in a five-hour stream, finding item #42 manually is a poor experience for both seller and consumer. Previous visual search approaches often fail because they rely solely on pixels, ignoring the rich auxiliary data—spoken descriptions, headlines, and real-time user comments—that provide vital context for identification.

The Fashion Focus Framework

Fashion Focus is an end-to-end system designed to automate product identification by fusing visual tracklets with linguistic data. The framework consists of two primary stages:

  1. Video Content Structuring: Instead of treating every frame independently, the system uses a joint detection and tracking approach with a DLA-34 backbone. It generates "tracklet proposals"—sequences of frames following a specific object—to ensure temporal consistency and focus on the product being exhibited rather than background noise.
  2. Multi-modal Retrieval: This core innovation extracts features from three distinct streams:
  3. Visual Stream: CNN features derived from the tracklets.
  4. Linguistic Stream: Audio processed via Automatic Speech Recognition (ASR) to extract descriptive keywords.
  5. Interaction Stream: Real-time user comments (e.g., "Can you show item #9?") and headlines mined for contextual clues.
The Fashion Focus architecture, illustrating the transition from raw video streams to multi-modal embeddings for product retrieval.

These modalities are fused into a compact embedding using LXMERT, a transformer-based model for learning vision-and-language representations. The system then performs retrieval by matching these multi-modal video embeddings against a database of static product images using triplet loss metric learning.

Experimental Evaluation

The researchers evaluated the system using a large-scale dataset from the Taobaolive platform, comprising 1,000 live-stream videos (5–7 hours each) across categories like clothing, shoes, bags, and beauty products. Ground truth was established using manually matched product-to-video timestamps.

The study compared three methods to measure the "Recall" of correctly localized commodities:

  • Visual Image (Vis_Img): 12.5% mean recall.
  • Visual Tracklet (Vis_Trk): 14.7% mean recall, demonstrating the value of temporal tracking.
  • Multi-modal Tracklet (Mlt_Trk): 16.8% mean recall.

The multi-modal approach provided a significant boost across all categories, with the highest performance in Clothing (23.6% recall) and Shoes (19.2% recall), where visual and verbal descriptions tend to be the most descriptive.

Conclusion and Future Work

Fashion Focus demonstrates that context acts as a powerful "weak annotation." By leveraging audio and comments, the system narrows the search space more effectively than visual features alone. Using tracklets instead of static frames provides robustness against occlusion and blur, allowing the model to "see" the product from multiple angles as the host moves it.

While a 16.8% recall marks a significant improvement over baseline visual search, fully autonomous video-to-shop matching remains a difficult frontier. The authors note that "recognition at scale" still faces challenges with the long-tail distribution of products. Future work will focus on refining multi-modal fusion—perhaps by weighing the reliability of noisy comments—and further reducing latency for massive-scale, real-time streaming applications.

Summary updated

Abstract

Nowadays, live-stream and short video shopping in E-commerce have grown exponentially. However, the sellers are required to manually match images of the selling products to the timestamp of exhibition in the untrimmed video, resulting in a complicated process. To solve the problem, we present an innovative demonstration of multi-modal retrieval system called ``Fashion Focus'', which enables to exactly localize the product images in the online video as the focuses. Different modality contributes to the community localization, including visual content, linguistic features and interaction context are jointly investigated via presented multi-modal learning. Our system employs two procedures for analysis, including video content structuring and multi-modal retrieval, to automatically achieve accurate video-to-shop matching. Fashion Focus presents a unified framework that can orientate the consumers towards relevant product exhibitions during watching videos and help the sellers to effectively deliver the products over search and recommendation.
BibTeX
@inproceedings{zhang2021fashion,
  author = {Yanhao Zhang and Qiang Wang and Pan Pan and Yun Zheng and Cheng Da and Siyang Sun and Yinghui Xu},
  title = {{Fashion Focus: Multi-modal Retrieval System for Video Commodity Localization in E-commerce}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=NxcZthKV5D},
}
Figure