Abstract
Learning a good representation of text is key to many recommendation applications. Examples include news recommendation where texts to be recommended are constantly published everyday. However, most existing recommendation techniques, such as matrix factorization based methods, mainly rely on interaction histories to learn representations of items. While latent factors of items can be learned effectively from user interaction data, in many cases, such data is not available, especially for newly emerged items. In this work, we aim to address the problem of personalized recommendation for completely new items with text information available. We cast the problem as a personalized text ranking problem and propose a general framework that combines text embedding with personalized recommendation. Users and textual content are embedded into latent feature space. The text embedding function can be learned end-to-end by predicting user interactions with items. To alleviate sparsity in interaction data, and leverage large amount of text data with little or no user interactions, we further propose a joint text embedding model that incorporates unsupervised text embedding with a combination module. Experimental results show that our model can significantly improve the effectiveness of recommendation systems on real-world datasets.
Introduction
Modern recommendation systems rely heavily on Collaborative Filtering (CF) techniques like matrix factorization, which learn from historical user-item interactions. However, CF fails completely for newly published items that have zero interaction history—a scenario incredibly common in news feeds and academic paper recommendations. To solve this "cold-start" problem, systems must look at the content of the new item. Traditional content-based methods usually represent text as a "bag-of-words" (like TF-IDF), which ignores word order, struggles with synonyms, and fails to capture deep semantic meaning. This paper proposes a framework to leverage modern deep learning text representations to recommend brand-new text items based on a user's historical preferences.
Methodology
The authors propose a framework called TER (Text Embedding for content-based Recommendation), which embeds both users and texts into a shared vector space where a simple dot product determines the recommendation score. The model is trained end-to-end using a pairwise ranking loss, learning to rank items a user clicked higher than those they didn't.
The framework evolved into Joint Text Embedding (TER+). Because user interaction data is often sparse, the authors introduce an unsupervised text embedding module. They pre-train document vectors using Paragraph Vector on massive, unlabeled text corpora. To fuse these signals, the model concatenates the supervised text vector ($h_1$) and the unsupervised pre-trained vector ($h_2$), passing them through a dense neural network layer.
A key component is the Combination Module. It applies dropout specifically to the unsupervised vector $h_2$. This acts as a regularization dial, controlling how much the model relies on general pre-trained knowledge versus task-specific user interaction data.
Experimental Setup
The models were evaluated on two real-world datasets:
- CiteULike: User-bookmarking data for academic papers (5.5k users, 16.9k items). An additional 339k DBLP papers were used for unsupervised pre-training.
- Yahoo! News Feed: User click data on news articles (10k users, 58.5k items). An additional 3.9 million news articles were used for unsupervised pre-training.
To strictly test the cold-start scenario, 20% of the items were held out as a test pool. All user interactions with these items were hidden during training.
Results and Discussion
The joint model (TER+) consistently outperformed all baselines, including Cosine Similarity, Multitask Logistic Regression, and Collaborative Deep Learning (CDL). On the CiteULike (title & abstract) dataset, TER+ (MoV) achieved a Mean Average Precision (MAP) of 0.7641, compared to CDL's 0.7484 and traditional Cosine matching's 0.7116.
A significant finding was the "semantic shift" between supervised and unsupervised learning. Unsupervised embeddings capture general semantics, while supervised embeddings capture domain-specific user intent. For example, the word "neural" shifted from general computer science associations in the unsupervised model to biology/neuroscience associations when trained on CiteULike clicks. Combining both allows the model to understand a richer, multi-faceted meaning of the text.
The use of dropout in the combination module proved crucial. By randomly zeroing out the unsupervised vector during training, the model was forced to learn robust supervised features while still benefiting from the general knowledge of the pre-trained corpus.
Interestingly, the simpler Mean of Vectors (MoV) embedding function frequently outperformed the more complex CNN, particularly on the CiteULike dataset. This suggests that for certain domains, specific keywords are more indicative of user interest than complex word ordering. Furthermore, the embedding approach showed a more significant performance gap over traditional TF-IDF baselines on short texts (titles) compared to long texts, as embeddings excel at matching concepts without exact keyword overlap.
Conclusion
The proposed TER+ framework effectively addresses the cold-start problem in text recommendation by combining supervised and unsupervised embeddings. While the combination module is currently a simple concatenation with dropout, future work will explore more sophisticated fusion techniques. The success of simple MoV embeddings also suggests that future research could refine CNN architectures to better capture the nuances of specific text domains.
Summary updated