Internal and Contextual Attention Network for Cold-start Multi-channel Matching in Recommendation

Ruobing Xie, Zhijie Qiu, Jun Rao, Yi Liu, Bo Zhang, Leyu Lin

Introduction: The Cold-Start Channel Challenge

In modern industrial recommendation systems, the standard architecture involves a two-stage pipeline: matching and ranking. Matching efficiently retrieves a few hundred candidates from millions of items, while ranking precisely orders them for the user. As platforms evolve, they often adopt multi-channel matching to handle diverse content types like news articles, short videos, and long-form videos simultaneously.

Overview of the multi-channel matching and ranking pipeline in an integrated recommendation system.

A significant hurdle arises when a new "cold-start" channel is introduced. Because these new channels lack historical user interaction data, traditional personalized matching fails. For instance, while a system might know a user's preference for sports articles, it has no data on their short video interests. Bridging this information gap is difficult because different content types use different metadata (e.g., "word count" for articles vs. "visual tags" for videos).

The ICAN Framework

To address this, researchers at Tencent developed the Internal and Contextual Attention Network (ICAN). ICAN leverages rich user history from mature, data-rich channels to inform retrieval in new, data-sparse channels.

The ICAN architecture, featuring contextual and internal attention layers to bridge information across multiple channels.

The framework consists of four primary components:

  1. Heterogeneous Feature Layer: This layer aggregates user behaviors across all channels, grouping them into standardized feature fields such as Item ID, Category, and Tag.
  2. Contextual Attention Layer: This layer dynamically weights the importance of historical feature fields based on the user's current target channel and context (e.g., network status). If a user is browsing short videos, the model highlights relevant tags from their article reading history.
  3. Internal Attention Layer: Utilizing a multi-head self-attention mechanism, this layer captures complex, non-linear interactions between feature fields across different channels. It learns how interests in one domain (e.g., "Cooking Articles") translate to another (e.g., "Food Preparation Videos").
  4. User Preference Representation: The final output is a dense user embedding optimized for high-speed Approximate Nearest Neighbor (ANN) search, ensuring $O(\log n)$ retrieval complexity during online serving.

Large-Scale Evaluation: HMM-700M

The researchers evaluated ICAN using a massive new dataset, HMM-700M, derived from WeChat Top Stories. This dataset contains nearly 700 million click behaviors from 30 million users across three channels: Articles, Long Videos (mature), and Short Videos (cold-start). It includes nine distinct feature fields, providing a robust testbed for multi-channel synergy.

Key Results and Production Impact

ICAN was tested both offline and in a live production environment on WeChat, showing substantial improvements over baselines like Youtube-DNN and DeepFM.

  • Offline Performance: In the cold-start "Short Video" channel, ICAN achieved a HIT@100 of 0.2132, a significant leap from the standard Youtube-DNN score of 0.1480.
  • Online A/B Testing: Deployment to millions of users resulted in a +27.03% Click-Through Rate (CTR) in the cold-start channel, a +15.84% increase in Average watched Category Number (ACN) for diversity, and a +5.60% gain in Average Reading Time (ART).

Conclusion and Future Directions

ICAN demonstrates that mature channels can act as "teachers" for new ones by focusing on field-level interactions rather than just item-level sequences. While highly effective, future work will explore automating feature field discovery and incorporating explicit transfer learning to handle even sparser data scenarios. The computational intensity of training on 700M+ records also presents an opportunity for developing more efficient training regimes.

Summary updated

Abstract

Real-world integrated personalized recommendation systems usually deal with millions of heterogeneous items. It is extremely challenging to conduct full corpus retrieval with complicated models due to the tremendous computation costs. Hence, most large-scale recommendation systems consist of two modules: a multi-channel matching module to efficiently retrieve a small subset of candidates, and a ranking module for precise personalized recommendation. However, multi-channel matching usually suffers from cold-start problems when adding new channels or new data sources. To solve this issue, we propose a novel Internal and contextual attention network (ICAN), which highlights channel-specific contextual information and feature field interactions between multiple channels. In experiments, we conduct both offline and online evaluations with case studies on a real-world integrated recommendation system. The significant improvements confirm the effectiveness and robustness of ICAN, especially for cold-start channels. Currently, ICAN has been deployed on WeChat Top Stories used by millions of users. The source code can be obtained from https://github.com/zhijieqiu/ICAN.
BibTeX
@inproceedings{xie2020internal,
  author = {Ruobing Xie and Zhijie Qiu and Jun Rao and Yi Liu and Bo Zhang and Leyu Lin},
  title = {{Internal and Contextual Attention Network for Cold-start Multi-channel Matching in Recommendation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/379},
}
Figure