TL;DR
This paper introduces a large-scale, privacy-aware dataset of Twitter Home Timeline engagements to address the lack of public benchmarks for recommender systems. They also detail the challenges and methodology behind the RecSys 2020 Challenge, which aims to spur research in predicting user engagement on social media.
What Problem Does This Paper Address?
Recommender systems are crucial for user satisfaction on social media, but predicting user engagement on platforms like Twitter, especially on the Home Timeline, has been limited by a lack of large, publicly available datasets. While Twitter data has been used for various analyses, the implicit feedback from user engagements (likes, replies, retweets) on their personalized timelines remains underexplored. This gap hinders the development and benchmarking of more powerful recommender models. Furthermore, ensuring user privacy while releasing such data is a significant challenge.
Methods
The paper outlines the creation of a large-scale dataset of Twitter Home Timeline engagements, focusing on privacy preservation. Key methods include:
- Privacy-Aware Data Release: Public tweets are used, and steps are taken to prevent de-anonymization, such as hashing user and tweet identifiers and creating "pseudo-negative" examples to avoid revealing user viewing history. Deleted content is also scrubbed from the dataset to maintain user choice.
- Engagement Prediction Task: The core task is framed as predicting user engagement with tweets, which can be approached as a binary classification or ranking problem.
- Feature Engineering: The dataset includes user features (follower count, verification status), tweet features (media types, links, language), and engagement features (timestamps of interactions).
- State-of-the-Art Models: The paper reviews various recommender system architectures, including Neural Collaborative Filtering (CF), Wide & Deep models, DeepFM, and Deep & Cross networks, highlighting their strengths and weaknesses in handling sparse data and complex interactions.
- Baseline Model: A simple baseline model is described, utilizing numeric, categorical, and ID features, with a tweet text embedding component. This model uses a multi-layer perceptron for prediction and is trained with Adam optimizer.
Datasets
The primary dataset introduced is a large-scale dataset of Twitter Home Timeline engagements, comprising approximately 160 million engagements sampled over one week, with an additional 40 million for validation and testing from the following week. This dataset is made available through the RecSys 2020 Challenge. It is designed to be privacy-preserving by using only public tweets and employing anonymization techniques.
Key Insights and Ideas
- Implicit Feedback is Key: User engagements (likes, replies, retweets) on the Home Timeline are a rich source of implicit feedback that can be leveraged for better recommendations, but this has been historically underutilized due to data limitations.
- Privacy is Paramount: Releasing large-scale social media data requires careful consideration of privacy. The paper demonstrates techniques like pseudo-negative sampling and scrubbing deleted content to balance utility and privacy.
- Multi-faceted Challenges: Predicting engagement on Twitter is complex due to issues like data shift, label imbalance, the scale of the social graph, and the informal nature of language.
- Benchmarking is Crucial: The release of a standardized, large-scale dataset and a challenge is vital for the research community to benchmark and advance recommender system techniques for social media.
Key Results
The paper does not present specific quantitative results from the challenge participants, as it focuses on defining the problem, challenges, and dataset. However, it does:
- Release a Large-Scale Dataset: The primary "result" is the provision of a dataset of 160 million engagements, which is claimed to be the largest public dataset released by a social network platform for this purpose.
- Describe a Baseline Model: A baseline model is detailed, providing a starting point for participants. The paper mentions training this baseline for 1 million steps with an Adam optimizer.
- Outline Evaluation Metrics: Two metrics are proposed for evaluating models: Relative Cross Entropy (RCE) and Area Under the Precision-Recall Curve (PR-AUC).
Limitations and Future Work
- Limitations:
- Pseudo-Negative Sampling: While designed for privacy, the creation of pseudo-negative examples might not perfectly represent true negative interactions (i.e., tweets the user saw but ignored).
- Dynamic Dataset: The dataset is continuously updated to reflect deleted content, which means it's not static, posing a challenge for strict reproducibility unless participants also update their local copies.
- Potential for De-anonymization: Despite efforts, the authors acknowledge that de-anonymization of the original public dataset might still be possible, though they aim to discourage misuse.
- Future Work:
- Contextual Information: Incorporating contextual information (e.g., time of day, user's current situation) can further enrich recommendation models.
- Beyond Engagement: Exploring other aspects of content relevance beyond simple engagement, such as serendipity, diversity, and coverage, is important for a holistic understanding of user satisfaction.
- Industry-Academia Collaboration: The paper emphasizes the need for continued collaboration between industry and academia to tackle complex challenges in recommender systems.
Summary updated