From Zero-Shot Learning to Cold-Start Recommendation

Jingjing Li, Mengmeng Jing, Ke Lu, Lei Zhu, Yang Yang, Zi Huang

In the landscape of machine learning, two distinct research communities have long grappled with a shared fundamental challenge: predicting outcomes for entities with no prior observation. Computer vision researchers face this in Zero-Shot Learning (ZSL), where models must identify unseen objects (e.g., a "Zebra") based solely on semantic descriptions (e.g., "striped," "horse-like"). Simultaneously, recommender system engineers confront the Cold-Start Recommendation (CSR) problem, attempting to suggest items to new users who possess no interaction history, relying only on profile attributes or social data.

This AAAI-19 paper by Li et al. formally bridges these two domains. By reformulating CSR as a ZSL problem, the authors introduce the Low-rank Linear Auto-Encoder (LLAE), a model that is both computationally efficient and highly effective at generalizing across sparse datasets.

Comparison of Zero-Shot Learning (ZSL) and Cold-Start Recommendation (CSR) frameworks.

The Cold-Start Challenge

Traditional collaborative filtering fails when a user has no interaction history, as there is no basis for calculating similarity. To mitigate this, systems incorporate "side information" such as age, location, or social connections. The authors argue that this process is functionally identical to ZSL: using an attribute space (semantic descriptions) to predict occurrences in a feature space (user behavior or image pixels).

However, existing CSR methods often suffer from three critical issues:

  1. Domain Shift: The statistical distribution of "warm" users (those with history) often differs significantly from "cold" users.
  2. Spurious Correlations: In sparse datasets, models may learn noise rather than genuine preferences.
  3. Efficiency: Real-world systems require models that can scale to millions of users with minimal latency.

Proposed Method: Low-rank Linear Auto-Encoder (LLAE)

To address these hurdles, the authors propose the LLAE. Unlike standard models that learn a one-way mapping, LLAE is designed as a bilateral system.

The architecture of the Low-rank Linear Auto-Encoder (LLAE).

Key Architectural Features

  • Bilateral Mapping: The model consists of an Encoder ($W$) that maps user behavior into the attribute space, and a Decoder ($W^T$) that reconstructs behavior from those attributes. By using "tied weights" (where the decoder is the transpose of the encoder), the model ensures that the learned attributes are truly representative of the underlying behavior, effectively mitigating domain shift.
  • Low-Rank Constraint: The mapping matrix $W$ is forced to be low-rank. This acts as a powerful filter, stripping away noise and forcing the model to capture dominant, shared latent attributes (e.g., general interest in "sports") rather than idiosyncratic, sparse interactions.
  • Denoising Robustness: During training, the model randomly zeros out 10% of the input data, forcing it to learn how to "fill in the blanks" and improving its ability to generalize to new users.
  • Computational Efficiency: The optimization problem for LLAE can be solved using the Sylvester equation. This allows the entire training process to be executed with remarkable speed; in environments like Matlab, the core optimization can be reduced to a single line of code.

Experimental Validation

The authors evaluated LLAE across two distinct domains to prove its versatility:

1. Zero-Shot Learning (Vision)

Using datasets like AwA (Animals with Attributes) and CUB (bird species), LLAE achieved an average accuracy of 73.85%. This significantly outperformed the previous state-of-the-art method, LESD, which averaged 70.02%.

2. Cold-Start Recommendation (Social/Web)

On recommendation tasks using Flickr, YouTube, and LastFM, LLAE consistently surpassed baselines like BPR-Map and LoCo. For instance, on the Flickr dataset, LLAE achieved a 39.25% mAP@100, a substantial lead over LoCo's 33.57%.

3. Performance and Speed

Beyond accuracy, the model's efficiency is a standout feature. On the AwA dataset, LLAE trained approximately 1,000 times faster than the Semantic Similarity Embedding (SSE) method, making it highly suitable for large-scale production environments.

Conclusion

The LLAE model demonstrates that the symmetry between encoding and decoding, combined with low-rank regularization, provides a robust solution to the sparsity and domain shift problems inherent in both ZSL and CSR. While the current linear model offers massive gains in speed and interpretability, the authors suggest that future work involving Deep Autoencoders could further capture complex, non-linear relationships while maintaining the bilateral insights established here.

Summary updated

Abstract

Zero-shot learning (ZSL) and cold-start recommendation (CSR) are two challenging problems in computer vision and recommender system, respectively. In general, they are independently investigated in different communities. This paper, however, reveals that ZSL and CSR are two extensions of the same intension. Both of them, for instance, attempt to predict unseen classes and involve two spaces, one for direct feature representation and the other for supplementary description. Yet there is no existing approach which addresses CSR from the ZSL perspective. This work, for the first time, formulates CSR as a ZSL problem, and a tailor-made ZSL method is proposed to handle CSR. Specifically, we propose a Low-rank Linear Auto-Encoder (LLAE), which challenges three cruxes, i.e., domain shift, spurious correlations and computing efficiency, in this paper. LLAE consists of two parts, a low-rank encoder maps user behavior into user attributes and a symmetric decoder reconstructs user behavior from user attributes. Extensive experiments on both ZSL and CSR tasks verify that the proposed method is a win-win formulation, i.e., not only can CSR be handled by ZSL models with a significant performance improvement compared with several conventional state-of-the-art methods, but the consideration of CSR can benefit ZSL as well.
BibTeX
@inproceedings{li2019from,
  author = {Jingjing Li and Mengmeng Jing and Ke Lu and Lei Zhu and Yang Yang and Zi Huang},
  title = {{From Zero-Shot Learning to Cold-Start Recommendation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33014189},
}
Figure