Ask the GRU: Multi-task Learning for Deep Text Recommendations

Trapit Bansal, David Belanger, Andrew McCallum

TL;DR

This paper introduces a deep learning approach to text-based recommendation that replaces traditional "bag-of-words" models with Gated Recurrent Units (GRUs). By capturing word order and using multi-task learning to predict item metadata (like tags), the authors significantly outperform state-of-the-art methods, particularly in "cold-start" scenarios where new items have no prior ratings.

What Problem Does This Paper Address?

Recommendation systems often struggle with two major hurdles: data sparsity and the cold-start problem.

Standard collaborative filtering (CF) relies on a matrix of user-item ratings. If an item is brand new (cold-start), there are no ratings to go on. To fix this, researchers use "hybrid" models that look at item content—usually text like abstracts or plot summaries. However, most existing hybrid models (like Collaborative Topic Regression) treat text as a "bag-of-words," ignoring word order. This means they can't distinguish between "this paper is about deep learning, not LDA" and "this paper is about LDA, not deep learning."

Furthermore, deep models are prone to overfitting on sparse rating data. The authors seek a way to build a text encoder that is both semantically sensitive and well-regularized.

Methods

The authors propose a neural architecture that maps text directly to latent factors (the mathematical "profiles" used to match users to items).

  1. The GRU Encoder: Instead of averaging word vectors, they use a two-layer Recurrent Neural Network (RNN). The first layer is bi-directional, reading the text forward and backward to capture context. The second layer's outputs are mean-pooled into a single vector representing the document.
  2. Hybrid Latent Factors: The final item representation is a combination of this GRU-based text encoding and a traditional item-specific embedding. In cold-start situations, the item-specific part is zeroed out, leaving the GRU to do all the work based solely on the text.
  3. Multi-Task Learning (MTL): To prevent the high-capacity GRU from overfitting, the authors add a secondary task: tag prediction. The network must simultaneously predict which user-provided tags (like "machine learning" or "optimization") apply to the paper. This forces the text encoder to learn "discriminative" features that are useful for both categorization and recommendation.

Datasets

The models were evaluated on two publicly available datasets from CiteULike, a platform where users save and tag scientific papers:

  • Citeulike-a: 5,551 users, 16,980 papers, and 3,629 unique tags.
  • Citeulike-t: A sparser dataset with 5,219 users, 25,975 papers, and 4,222 tags.

Key Insights and Ideas

  • Order Matters: The "aha moment" here is proving that sequential modeling (RNNs) provides a tangible boost over bag-of-words in recommendation. By understanding phrases rather than just counting words, the model captures more nuanced item similarities.
  • MTL as Regularization: Deep learning in recommendation is often hampered by the "extreme sparsity" of ratings. Using metadata prediction as a "side task" acts as a powerful regularizer, effectively "teaching" the model what the text is about before it tries to guess who will like it.
  • Interpretability: Using saliency maps (calculating the gradient of the prediction with respect to input words), the authors show the model ignores stop words and focuses on meaningful technical phrases like "maximum likelihood" or "iteratively reweighted least squares."

Key Results

The authors compared their GRU-MTL model against Collaborative Topic Regression (CTR) and a simpler word-embedding average model (Embed).

  • Cold-Start Dominance: In the cold-start scenario, the GRU-based models achieved up to a 34% relative improvement in Recall@50 over the previous state-of-the-art (CTR).
  • Warm-Start Gains: Even when ratings were available, the GRU-MTL model outperformed CTR-MTL, showing that better text representations help across the board.
  • The MTL Boost: Adding the multi-task tag prediction improved the GRU's recall by roughly 7-8% on both datasets, confirming that the auxiliary task helps the model generalize better.

Limitations and Future Work

  • User Modeling: While the paper goes deep on item representation, the user profiles are still standard latent vectors. The authors suggest that applying deep architectures to user-generated data (like their past clicks or tweets) is a logical next step.
  • Multi-Modality: The current work is text-centric. The authors note that the multi-task framework could be expanded to include other modalities, such as using Convolutional Neural Networks (CNNs) to incorporate movie posters or product images into the recommendation engine.
  • Scalability: RNNs are computationally more expensive than bag-of-words models. While they mention GPU parallelization, scaling this to millions of items in real-time remains a standard engineering challenge for these types of architectures.

Summary updated

Abstract

In a variety of application domains the content to be recommended to users is associated with text. This includes research papers, movies with associated plot summaries, news articles, blog posts, etc. Recommendation approaches based on latent factor models can be extended naturally to leverage text by employing an explicit mapping from text to factors. This enables recommendations for new, unseen content, and may generalize better, since the factors for all items are produced by a compactly-parametrized model. Previous work has used topic models or averages of word embeddings for this mapping. In this paper we present a method leveraging deep recurrent neural networks to encode the text sequence into a latent vector, specifically gated recurrent units (GRUs) trained end-to-end on the collaborative filtering task. For the task of scientific paper recommendation, this yields models with significantly higher accuracy. In cold-start scenarios, we beat the previous state-of-the-art, all of which ignore word order. Performance is further improved by multi-task learning, where the text encoder network is trained for a combination of content recommendation and item metadata prediction. This regularizes the collaborative filtering model, ameliorating the problem of sparsity of the observed rating matrix.
BibTeX
@inproceedings{bansal2016ask,
  author = {Trapit Bansal and David Belanger and Andrew McCallum},
  title = {{Ask the GRU: Multi-task Learning for Deep Text Recommendations}},
  booktitle = {RecSys},
  year = {2016},
  url = {https://openreview.net/forum?id=BHe4qVlY8Wc},
}

Loading related content...

Figure