Privacy-Preserving Recommender Systems Challenge on Twitter's Home Timeline

Luca Belli, Sofia Ira Ktena, Alykhan Tejani, Alexandre Lung-Yut-Fong, Frank Portman, Xiao Zhu, Yuanpu Xie, Akshay Gupta, Michael M. Bronstein, Amra Delic, Gabriele Sottocornola, Vito Walter Anelli, Nazareno Andrade, Jessie Smith, Wenzhe ShiarXiv 2020 · OpenReview

TL;DR

This paper introduces a large-scale, privacy-aware dataset of Twitter Home Timeline engagements to address the lack of public benchmarks for recommender systems. They also detail the challenges and methodology behind the RecSys 2020 Challenge, which aims to spur research in predicting user engagement on social media.

What Problem Does This Paper Address?

Recommender systems are crucial for user satisfaction on social media, but predicting user engagement on platforms like Twitter, especially on the Home Timeline, has been limited by a lack of large, publicly available datasets. While Twitter data has been used for various analyses, the implicit feedback from user engagements (likes, replies, retweets) on their personalized timelines remains underexplored. This gap hinders the development and benchmarking of more powerful recommender models. Furthermore, ensuring user privacy while releasing such data is a significant challenge.

Methods

The paper outlines the creation of a large-scale dataset of Twitter Home Timeline engagements, focusing on privacy preservation. Key methods include:

  • Privacy-Aware Data Release: Public tweets are used, and steps are taken to prevent de-anonymization, such as hashing user and tweet identifiers and creating "pseudo-negative" examples to avoid revealing user viewing history. Deleted content is also scrubbed from the dataset to maintain user choice.
  • Engagement Prediction Task: The core task is framed as predicting user engagement with tweets, which can be approached as a binary classification or ranking problem.
  • Feature Engineering: The dataset includes user features (follower count, verification status), tweet features (media types, links, language), and engagement features (timestamps of interactions).
  • State-of-the-Art Models: The paper reviews various recommender system architectures, including Neural Collaborative Filtering (CF), Wide & Deep models, DeepFM, and Deep & Cross networks, highlighting their strengths and weaknesses in handling sparse data and complex interactions.
  • Baseline Model: A simple baseline model is described, utilizing numeric, categorical, and ID features, with a tweet text embedding component. This model uses a multi-layer perceptron for prediction and is trained with Adam optimizer.

Datasets

The primary dataset introduced is a large-scale dataset of Twitter Home Timeline engagements, comprising approximately 160 million engagements sampled over one week, with an additional 40 million for validation and testing from the following week. This dataset is made available through the RecSys 2020 Challenge. It is designed to be privacy-preserving by using only public tweets and employing anonymization techniques.

Key Insights and Ideas

  • Implicit Feedback is Key: User engagements (likes, replies, retweets) on the Home Timeline are a rich source of implicit feedback that can be leveraged for better recommendations, but this has been historically underutilized due to data limitations.
  • Privacy is Paramount: Releasing large-scale social media data requires careful consideration of privacy. The paper demonstrates techniques like pseudo-negative sampling and scrubbing deleted content to balance utility and privacy.
  • Multi-faceted Challenges: Predicting engagement on Twitter is complex due to issues like data shift, label imbalance, the scale of the social graph, and the informal nature of language.
  • Benchmarking is Crucial: The release of a standardized, large-scale dataset and a challenge is vital for the research community to benchmark and advance recommender system techniques for social media.

Key Results

The paper does not present specific quantitative results from the challenge participants, as it focuses on defining the problem, challenges, and dataset. However, it does:

  • Release a Large-Scale Dataset: The primary "result" is the provision of a dataset of 160 million engagements, which is claimed to be the largest public dataset released by a social network platform for this purpose.
  • Describe a Baseline Model: A baseline model is detailed, providing a starting point for participants. The paper mentions training this baseline for 1 million steps with an Adam optimizer.
  • Outline Evaluation Metrics: Two metrics are proposed for evaluating models: Relative Cross Entropy (RCE) and Area Under the Precision-Recall Curve (PR-AUC).

Limitations and Future Work

  • Limitations:
    • Pseudo-Negative Sampling: While designed for privacy, the creation of pseudo-negative examples might not perfectly represent true negative interactions (i.e., tweets the user saw but ignored).
  • Dynamic Dataset: The dataset is continuously updated to reflect deleted content, which means it's not static, posing a challenge for strict reproducibility unless participants also update their local copies.
  • Potential for De-anonymization: Despite efforts, the authors acknowledge that de-anonymization of the original public dataset might still be possible, though they aim to discourage misuse.
  • Future Work:
    • Contextual Information: Incorporating contextual information (e.g., time of day, user's current situation) can further enrich recommendation models.
  • Beyond Engagement: Exploring other aspects of content relevance beyond simple engagement, such as serendipity, diversity, and coverage, is important for a holistic understanding of user satisfaction.
  • Industry-Academia Collaboration: The paper emphasizes the need for continued collaboration between industry and academia to tackle complex challenges in recommender systems.

Summary updated

Abstract

Recommender systems constitute the core engine of most social network platforms nowadays, aiming to maximize user satisfaction along with other key business objectives. Twitter is no exception. Despite the fact that Twitter data has been extensively used to understand socioeconomic and political phenomena and user behaviour, the implicit feedback provided by users on Tweets through their engagements on the Home Timeline has only been explored to a limited extent. At the same time, there is a lack of large-scale public social network datasets that would enable the scientific community to both benchmark and build more powerful and comprehensive models that tailor content to user interests. By releasing an original dataset of 160 million Tweets along with engagement information, Twitter aims to address exactly that. During this release, special attention is drawn on maintaining compliance with existing privacy laws. Apart from user privacy, this paper touches on the key challenges faced by researchers and professionals striving to predict user engagements. It further describes the key aspects of the RecSys 2020 Challenge that was organized by ACM RecSys in partnership with Twitter using this dataset.
BibTeX
@misc{belli2020privacy,
  author = {Luca Belli and Sofia Ira Ktena and Alykhan Tejani and Alexandre Lung-Yut-Fong and Frank Portman and Xiao Zhu and Yuanpu Xie and Akshay Gupta and Michael M. Bronstein and Amra Delic and Gabriele Sottocornola and Vito Walter Anelli and Nazareno Andrade and Jessie Smith and Wenzhe Shi},
  title = {{Privacy-Preserving Recommender Systems Challenge on Twitter's Home Timeline}},
  year = {2020},
  eprint = {2004.13715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.13715},
}
Figure