Visual Language - Let the Product Say What You Want

Jiaying Wang, Shuailing Hao, Jing Shan, Xiaoxu SongAAAI 2024 · DOI

Abstract

The "Visual Language" system is a multitasking framework designed to streamline e-commerce operations. It automatically generates fine-grained product descriptions for sellers and provides high-precision multimodal retrieval for buyers. By integrating a specialized product knowledge base with pre-trained models like CLIP and GPT-2, the system bridges the gap between raw visual data and nuanced semantic labels such as "Nordic style" or "soft decoration."

Introduction

In the e-commerce sector, two significant friction points persist: sellers often struggle to produce accurate and appealing product descriptions efficiently, while customers face challenges searching for specific items using vague text or images. Existing vision-language models frequently fall short in retail contexts because they treat images as monolithic entities, often focusing on background elements rather than the specific product. Furthermore, standard models typically lack the granularity required to identify specific market-defining aesthetics, such as "Nordic minimalist."

Methodology

The proposed "Visual Language" architecture consists of three primary stages designed to ensure high-fidelity translation from visual input to semantic output.

The complete Visual Language architecture, spanning feature encoding, mode mapping, and inference.

1. Feature Encoder & Product Label Generation

Rather than relying solely on raw pixel data, the system utilizes CLIP to extract image features, which are then matched against a custom Knowledge Base (KB). This KB, constructed using Vision Transformers (ViT) and high-quality phrase mining, provides fine-grained semantic labels (e.g., "creative pillows"). By calculating the similarity between image features and these labels, the system filters out background noise to focus on core product attributes.

2. Mode Mapping Network

To harmonize visual features and text labels, the system employs a spatial mapping approach. BERT is used to fuse multi-dimensional product labels with image features, ensuring the resulting "fused features" reside in the same mathematical space as the language model's embeddings.

3. Inference and Retrieval

For generation tasks, fused features are fed into GPT-2 as a "visual prerequisite," guiding the model to produce descriptions relevant to specific product characteristics. For retrieval, a multimodal encoder and a K-Nearest Neighbors (KNN) algorithm allow for rapid matching of user queries against the knowledge base.

Key Innovations

A central innovation of this framework is the Knowledge Base Buffer. By inserting a Product Knowledge Graph layer, the system requires the AI to identify specific product attributes before generating sentences. Additionally, the Spatial Alignment approach prevents common "hallucinations" by ensuring visual inputs are smoothly integrated into the language model's processing pipeline. This multitasking synergy ensures a consistent vocabulary across both the selling (generation) and buying (retrieval) sides of the ecosystem.

Results and Demonstration

The framework demonstrates significant qualitative success in generating stylistically relevant descriptions for diverse categories, such as furniture and apparel.

Demonstration of the system generating product descriptions from both static images and video frames.

The system is versatile, supporting both static images and video content. For video, it processes individual frames and fuses their features into a coherent narrative. Retrieval capabilities are equally robust, allowing users to find products accurately using either text descriptions (e.g., "black leather boots") or reference images.

Conclusion and Future Work

While the "Visual Language" system offers improved detail over general-purpose models, future development will focus on enhancing description granularity further. A key objective is the identification of "unique geographic indications," enabling the AI to recognize regional product specialties. Subsequent research will also involve quantitative benchmarking using standard metrics like BLEU or METEOR to further validate performance against state-of-the-art models.

Summary updated

BibTeX
@inproceedings{wang2024visual,
  author = {Jiaying Wang and Shuailing Hao and Jing Shan and Xiaoxu Song},
  title = {{Visual Language - Let the Product Say What You Want}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30583},
}
Figure