Telepath: Understanding Users from a Human Vision Perspective in Large-Scale Recommender Systems

Yu Wang, Jixing Xu, Aohan Wu, Mantian Li, Yang He, Jinghe Hu, Weipeng P. YanAAAI 2018 · DOI

Introduction

Traditional recommender systems often treat items as abstract IDs (e.g., "Item #12345"). While effective for established products, this approach suffers from two major flaws: the "cold start" problem, where new items lack historical data, and a fundamental disregard for visual appeal. Humans are inherently visual shoppers; we are often drawn to specific colors, patterns, or styles that an ID-based system cannot capture.

Telepath, developed by JD.com, is a "bionic" recommender system designed to solve these issues by simulating human vision and brain activity. By using deep learning to "see" item images and analyze browsing sequences, Telepath has delivered significant gains in sales and engagement in one of the world's largest e-commerce environments.

The Telepath Architecture

The Telepath system utilizes a multi-module deep learning architecture designed to mimic the human decision-making process during shopping. It consists of three primary components:

  1. Vision Extraction Module (The "Eye"): Instead of relying on IDs, Telepath uses a custom Convolutional Neural Network (CNN) to process item images. This converts every product into a 50-dimensional "vision activation" vector, allowing the system to understand brand-new items as easily as established ones.
  2. Interest Understanding Module (The "Cortex"): This module simulates how the brain processes visual stimuli. It employs an RNN (LSTM) to process browsing history in reverse order to capture immediate "conscious" intent, and a DNN to process history in forward order to capture long-term "subconscious" preferences.
  3. Scoring Module (The "Decision Maker"): A "Wide & Deep" sub-network combines visual signals and interest vectors with traditional demographic data to predict the final ranking score.
The Telepath model architecture, illustrating the Vision Extraction, Interest Understanding, and Scoring modules.

Key Insights and Semantic Clustering

One of the most significant findings in the development of Telepath is the power of the reverse sequence. The authors discovered that LSTMs perform better when browsing history is fed in reverse, as items clicked minutes ago are more relevant to current intent than those clicked days ago.

Furthermore, the model accounts for "visual curiosity," acknowledging that users may click items simply because they are visually attractive. Qualitative analysis using t-SNE visualizations demonstrated that the CNN naturally groups items by visual style and category without explicit labels. For instance, items with similar cartoon packaging or specific aesthetics are clustered together, proving the model understands "attractiveness" beyond simple product categories.

t-SNE visualization of item embeddings showing semantic clustering of product categories like cellphones, cookies, and beach shoes.

Real-World Performance

Telepath was deployed in JD.com’s live production environment, yielding impressive results across various platforms:

  • JD App Recommendations: Increased Click-Through Rate (CTR) by 1.59%, Gross Merchandise Value (GMV) by 8.16%, and total orders by 8.71%.
  • Advertising (JD DSP): The initial launch saw a 61.72% increase in GMV and a 65.57% increase in Return on Investment (ROI).
  • Efficiency: The system demonstrated that a model using only images could achieve an Area Under the Curve (AUC) comparable to traditional models using billions of ID-based weights.

Conclusion and Future Work

While Telepath revolutionizes recommendations through visual understanding, it currently focuses primarily on images. The authors acknowledge that textual descriptions (size, material, etc.) are also vital signals. Future work involves integrating a text-understanding subnetwork using character-level CNNs and applying the Telepath framework to the initial "retrieval" stage to improve the diversity of candidate items before they reach the ranking stage.

Summary updated

BibTeX
@inproceedings{wang2018telepath,
  author = {Yu Wang and Jixing Xu and Aohan Wu and Mantian Li and Yang He and Jinghe Hu and Weipeng P. Yan},
  title = {{Telepath: Understanding Users from a Human Vision Perspective in Large-Scale Recommender Systems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11243},
}
Figure