Ask the GRU: Multi-Task Learning for Deep Text Recommendations

Trapit Bansal, David Belanger, Andrew McCallum

Abstract

In a variety of application domains the content to be recommended to users is associated with text. This includes research papers, movies with associated plot summaries, news articles, blog posts, etc. Recommendation approaches based on latent factor models can be extended naturally to leverage text by employing an explicit mapping from text to factors. This enables recommendations for new, unseen content, and may generalize better, since the factors for all items are produced by a compactly-parametrized model. Previous work has used topic models or averages of word embeddings for this mapping. In this paper we present a method leveraging deep recurrent neural networks to encode the text sequence into a latent vector, specifically gated recurrent units (GRUs) trained end-to-end on the collaborative filtering task. For the task of scientific paper recommendation, this yields models with significantly higher accuracy. In cold-start scenarios, we beat the previous state-of-the-art, all of which ignore word order. Performance is further improved by multi-task learning, where the text encoder network is trained for a combination of content recommendation and item metadata prediction. This regularizes the collaborative filtering model, ameliorating the problem of sparsity of the observed rating matrix.

Introduction

Recommendation systems frequently encounter two significant challenges: data sparsity and the cold-start problem. Standard collaborative filtering (CF) relies on a matrix of user-item ratings; however, when an item is new (cold-start), no ratings exist to inform recommendations. Hybrid models address this by incorporating item content, such as text abstracts. Traditional hybrid models, like Collaborative Topic Regression (CTR), typically treat text as a "bag-of-words," which ignores word order and semantic nuances. For instance, a bag-of-words model cannot distinguish between "this paper is about deep learning, not LDA" and "this paper is about LDA, not deep learning." Furthermore, deep models are prone to overfitting on sparse rating data. This research proposes a text encoder that is both semantically sensitive and well-regularized through multi-task learning.

Methodology

The proposed architecture maps text directly to latent factors using a neural network that captures sequential dependencies.

The GRU Encoder

Instead of simple word vector averaging, the model employs a two-layer Recurrent Neural Network (RNN). The first layer is bi-directional, processing text in both directions to capture context. The outputs of the second layer are mean-pooled into a single vector representing the document. This allows the model to understand complex phrases rather than just counting words.

Hybrid Latent Factors and Multi-Task Learning (MTL)

The final item representation combines this GRU-based text encoding with a traditional item-specific embedding. In cold-start scenarios, the item-specific component is zeroed, relying entirely on the text encoder. To prevent overfitting on sparse rating data, the authors introduce a secondary task: tag prediction. The network is trained to simultaneously predict user-provided tags (e.g., "RNN", "LDA"), forcing the encoder to learn discriminative features useful for both recommendation and categorization.

The GRU-MTL architecture showing the bi-directional GRU layers, pooling, and the multi-task output for user ratings (r) and tag prediction (p).

Experimental Results

The models were evaluated on two datasets from CiteULike: Citeulike-a (5,551 users, 16,980 papers) and Citeulike-t (5,219 users, 25,975 papers). The GRU-MTL model was compared against Collaborative Topic Regression (CTR) and a word-embedding average model (Embed).

Recall@50 performance comparison across Citeulike-a and Citeulike-t datasets for warm-start, cold-start, and tag prediction tasks.

Key findings include:

  • Cold-Start Dominance: In cold-start scenarios, GRU-based models achieved up to a 34% relative improvement in Recall@50 over the previous state-of-the-art (CTR).
  • Warm-Start Gains: GRU-MTL outperformed CTR-MTL even when ratings were available, indicating that superior text representations benefit the model generally.
  • MTL as Regularization: Adding the auxiliary tag prediction task improved the GRU's recall by approximately 7-8% on both datasets, confirming its role as a powerful regularizer against rating sparsity.
  • Interpretability: Using saliency maps, the authors observed that the model ignores stop words and focuses on meaningful technical phrases like "maximum likelihood."

Conclusion

This work demonstrates that capturing word order via GRUs significantly improves text-based recommendations. Multi-task learning effectively mitigates the challenges of rating sparsity by providing an auxiliary training signal. Future research could extend this framework to deep user modeling or incorporate multi-modal data such as images. While RNNs present higher computational costs than bag-of-words models, their performance gains, particularly in cold-start settings, make them a compelling choice for modern recommendation engines.

Summary updated

Abstract

In a variety of application domains the content to be recommended to users is associated with text. This includes research papers, movies with associated plot summaries, news articles, blog posts, etc. Recommendation approaches based on latent factor models can be extended naturally to leverage text by employing an explicit mapping from text to factors. This enables recommendations for new, unseen content, and may generalize better, since the factors for all items are produced by a compactly-parametrized model. Previous work has used topic models or averages of word embeddings for this mapping. In this paper we present a method leveraging deep recurrent neural networks to encode the text sequence into a latent vector, specifically gated recurrent units (GRUs) trained end-to-end on the collaborative filtering task. For the task of scientific paper recommendation, this yields models with significantly higher accuracy. In cold-start scenarios, we beat the previous state-of-the-art, all of which ignore word order. Performance is further improved by multi-task learning, where the text encoder network is trained for a combination of content recommendation and item metadata prediction. This regularizes the collaborative filtering model, ameliorating the problem of sparsity of the observed rating matrix.
BibTeX
@inproceedings{bansal2016ask,
  author = {Trapit Bansal and David Belanger and Andrew McCallum},
  title = {{Ask the GRU: Multi-Task Learning for Deep Text Recommendations}},
  booktitle = {RecSys},
  year = {2016},
  url = {https://openreview.net/forum?id=BHe4qVlY8Wc},
}
Figure