Fashion Focus: Multi-modal Retrieval System for Video Commodity Localization in E-commerce

Yanhao Zhang, Qiang Wang, Pan Pan, Yun Zheng, Cheng Da, Siyang Sun, Yinghui Xu

Abstract

Nowadays, live-stream and short video shopping in E-commerce have grown exponentially. However, the sellers are required to manually match images of the selling products to the timestamp of exhibition in the untrimmed video, resulting in a complicated process. To solve the problem, we present an innovative demonstration of multi-modal retrieval system called "Fashion Focus", which enables to exactly localize the product images in the online video as the focuses. Different modality contributes to the community localization, including visual content, linguistic features and interaction context are jointly investigated via presented multi-modal learning. Our system employs two procedures for analysis, including video content structuring and multi-modal retrieval, to automatically achieve accurate video-to-shop matching. Fashion Focus presents a unified framework that can orientate the consumers towards relevant product exhibitions during watching videos and help the sellers to effectively deliver the products over search and recommendation.

Introduction

The explosion of live-stream shopping has introduced a significant metadata bottleneck for e-commerce platforms. Currently, sellers must manually synchronize product listings with specific timestamps in hours-long videos to ensure "buy" links appear correctly. This manual process is tedious and prone to errors. From a technical standpoint, this "video-to-shop" retrieval task is complicated by dynamic environments, a long-tail distribution of millions of similar fashion items, and the need for real-time processing at scale.

The Fashion Focus Framework

To address these challenges, the "Fashion Focus" system utilizes a two-stage framework designed to automate the matching of video segments to product images.

The Fashion Focus framework overview, illustrating video content structuring and multi-modal retrieval.

1. Video Content Structuring

Rather than analyzing isolated frames, the system generates tracklet proposals. Using a one-stage video detector with a DLA-34 backbone, the system jointly performs object detection and appearance embedding. This allows it to follow a specific item across multiple frames, creating a "tracklet" that provides a more robust visual representation than a single image. A hierarchical classification approach is employed to mitigate category ambiguity in the long-tail distribution.

2. Multi-modal Retrieval and Fusion

The core of the system is the fusion of disparate data streams to accurately "triangulate" products. The system integrates:

  • Visual Features: Extracted directly from the tracklets.
  • Audio Data: Speech is converted to text via Automatic Speech Recognition (ASR).
  • Linguistic/Contextual Data: User comments (e.g., "Can you show item No. 9?") and video headlines are mined for cues.

These modalities are fused using the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) model. This maps visual and linguistic features into a single compact embedding space, optimized using a triplet constraint to ensure matching video segments and product images are mathematically close.

Experimental Evaluation

The system was evaluated on a large-scale dataset from Alibaba (Taobao), consisting of 1,000 live-stream videos (5–7 hours each) across categories like clothing, shoes, and beauty products. The primary metric was the recall of identical matched products.

Visualizations of tracklet proposals and the user interface for focus exhibition and commodity delivery.

The results demonstrate that the multi-modal tracklet approach (Mlt_Trk) significantly outperforms baseline visual-only methods:

Method Clothing Shoe Bag Mean Recall
Vis_Img (Single Image) 16.4% 15.1% 13.7% 12.5%
Vis_Trk (Visual Tracklet) 19.3% 17.1% 16.4% 14.7%
Mlt_Trk (Multi-modal) 23.6% 19.2% 18.3% 16.8%

The inclusion of temporal tracking (Vis_Trk) improves performance over static images, but the multi-modal integration provides the most substantial boost, particularly in the clothing category where visual ambiguity is highest.

Conclusion

Fashion Focus demonstrates that "noisy" live-stream data, such as audio explanations and user comments, can serve as valuable "weak annotations" to improve retrieval accuracy. While a mean recall of 16.8% highlights the ongoing difficulty of automated video-to-shop matching, the system provides a scalable foundation for reducing manual labor for sellers and enhancing the consumer shopping experience. Future work will focus on improving real-time efficiency and addressing the "vanishing-classification" problem for rare items.

Summary updated

Abstract

Nowadays, live-stream and short video shopping in E-commerce have grown exponentially. However, the sellers are required to manually match images of the selling products to the timestamp of exhibition in the untrimmed video, resulting in a complicated process. To solve the problem, we present an innovative demonstration of multi-modal retrieval system called "Fashion Focus", which enables to exactly localize the product images in the online video as the focuses. Different modality contributes to the community localization, including visual content, linguistic features and interaction context are jointly investigated via presented multi-modal learning. Our system employs two procedures for analysis, including video content structuring and multi-modal retrieval, to automatically achieve accurate video-to-shop matching. Fashion Focus presents a unified framework that can orientate the consumers towards relevant product exhibitions during watching videos and help the sellers to effectively deliver the products over search and recommendation.
BibTeX
@inproceedings{zhang2021fashion,
  author = {Yanhao Zhang and Qiang Wang and Pan Pan and Yun Zheng and Cheng Da and Siyang Sun and Yinghui Xu},
  title = {{Fashion Focus: Multi-modal Retrieval System for Video Commodity Localization in E-commerce}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=NxcZthKV5D},
}
Figure