Image Matters: Jointly Train Advertising CTR Model with Image Representation of Ad and User Behavior

Tiezheng Ge, Liqin Zhao, Guorui Zhou, Keyu Chen, Shuying Liu, Huiming Yi, Zelin Hu, Bochao Liu, Peng Sun, Haoyu Liu, Pengtao Yi, Sui Huang, Zhiqiang Zhang, Xiaoqiang Zhu, Yu Zhang, Kun Gai

Abstract

In Taobao, the largest e-commerce platform in China, billions of items are typically displayed with their images. For better user experience and business effectiveness, Click Through Rate (CTR) prediction in online advertising systems exploits abundant user historical behaviors. Enhancing behavior representations with user behavior images helps understand a user's visual preference and improves CTR prediction accuracy significantly. We propose to model user preference jointly with user behavior ID features and behavior images. However, training with user behavior images presents challenges in communication and computation. To handle these, we propose the Advanced Model Server (AMS), a distributed machine learning paradigm that learns a unified image descriptor model shared by all server nodes, embedding large images into low-dimensional features before transmission. Based on AMS, we propose a Deep Image CTR Model (DICM). Our approach achieves significant improvements in both online and offline evaluations and has been deployed in Taobao's display advertising system.

TL;DR

Researchers at Alibaba have developed a way to significantly boost Click-Through Rate (CTR) prediction by incorporating the visual history of a user—the images of products they’ve clicked on—into the model. To make this computationally feasible at scale, they introduced the Advanced Model Server (AMS), which processes images on the server side to reduce communication costs by over 30x.

The Challenge: Visual Context in CTR Prediction

In e-commerce, predicting whether a user will click an ad is critical. Traditionally, these models rely on ID features (e.g., UserID, ItemID). However, IDs are "cold": for new items or users with rare tastes, the model struggles due to data sparsity.

Images offer a solution by providing intrinsic semantic information—a "blue silk dress" is visually identifiable regardless of its ID. While previous research used the ad's image, this paper tackles the harder problem of modeling the user’s visual behavior history. A typical user might have clicked on hundreds of items; processing all those images for every single prediction creates a massive bottleneck in storage, computation, and network communication.

Taobao ad display and the standard recommendation pipeline.

Methodology: Scaling Visual Behavior Modeling

The authors introduce two major innovations to bridge the gap between computer vision and recommendation systems:

1. Advanced Model Server (AMS)

In a standard Parameter Server (PS) architecture, worker nodes perform heavy computation while servers store and update parameters. AMS shifts this paradigm. Servers run a "sub-model" (an image descriptor) to process raw images into low-dimensional embeddings before sending them to workers. This reduces the communication load by 32x and storage requirements by 31x, making joint training feasible.

2. Deep Image CTR Model (DICM)

DICM is the neural network architecture designed to utilize these images. It employs a Multi-Query Attentive Pooling mechanism. The model uses the current ad's visual and ID features as a "query" to scan the user's history, assigning higher weights to past images that are visually or categorically relevant.

Architectures of the Deep Image CTR Model (DICM) and the Advanced Model Server (AMS).

3. Hybrid Training

To balance accuracy and speed, the first 14 layers of a pre-trained VGG16 are used as a fixed feature extractor, followed by a trainable 3-layer fully connected network that fine-tunes the image representation for the CTR task.

Key Insights and Results

  • Visual-ID Synergy: The biggest gains come from the interaction between images and IDs. The attention mechanism allows the model to "notice" visual similarities (e.g., same style of T-shirt) even if the brands or price points differ.
  • Efficiency: The AMS framework allowed training on billions of samples in just 18 hours using a 20-GPU cluster.
  • Performance: DICM outperformed the state-of-the-art ID-only baseline by 0.0078 in GAUC. In live A/B testing on Taobao, the model delivered a 9.2% increase in CTR and a 5.7% increase in platform revenue (eCPM).

Limitations and Future Work

The authors currently freeze early VGG16 layers to save computation, but acknowledge that end-to-end training might yield better results. Future work could extend AMS to handle multi-modal inputs like text and video, and explore applying visual models to the initial "Matching" (retrieval) stage of the ad pipeline.

Summary updated

Abstract

In Taobao, the largest e-commerce platform in China, billions of items are provided and typically displayed with their images. For better user experience and business effectiveness, Click Through Rate (CTR) prediction in online advertising system exploits abundant user historical behaviors to identify whether a user is interested in a candidate ad. Enhancing behavior representations with user behavior images will help understand user's visual preference and improve the accuracy of CTR prediction greatly. So we propose to model user preference jointly with user behavior ID features and behavior images. However, training with user behavior images brings tens to hundreds of images in one sample, giving rise to a great challenge in both communication and computation. To handle these challenges, we propose a novel and efficient distributed machine learning paradigm called Advanced Model Server (AMS). With the well known Parameter Server (PS) framework, each server node handles a separate part of parameters and updates them independently. AMS goes beyond this and is designed to be capable of learning a unified image descriptor model shared by all server nodes which embeds large images into low dimensional high level features before transmitting images to worker nodes. AMS thus dramatically reduces the communication load and enables the arduous joint training process. Based on AMS, the methods of effectively combining the images and ID features are carefully studied, and then we propose a Deep Image CTR Model. Our approach is shown to achieve significant improvements in both online and offline evaluations, and has been deployed in Taobao display advertising system serving the main traffic.
BibTeX
@inproceedings{ge2017image,
  author = {Tiezheng Ge and Liqin Zhao and Guorui Zhou and Keyu Chen and Shuying Liu and Huiming Yi and Zelin Hu and Bochao Liu and Peng Sun and Haoyu Liu and Pengtao Yi and Sui Huang and Zhiqiang Zhang and Xiaoqiang Zhu and Yu Zhang and Kun Gai},
  title = {{Image Matters: Jointly Train Advertising CTR Model with Image Representation of Ad and User Behavior}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2017},
}
Figure