TL;DR
This paper introduces a simple yet effective method to enrich user embeddings from social media by incorporating non-textual metadata like usernames and locations. By treating these as "dummy" tokens within existing text embedding models, they achieve significant improvements in friend recommendation tasks, demonstrating the value of jointly modeling diverse user information.
What Problem Does This Paper Address?
Social media platforms generate a wealth of data about users, including their posts (text), who they follow (network), and where they are (location). Representing users with dense, low-dimensional vectors (embeddings) is crucial for many applications like recommendation systems and user profiling. While previous work has explored combining different data types for user embeddings, adapting existing models to handle multiple "views" of data can be complex. This paper tackles the challenge of efficiently integrating diverse user metadata (beyond just text) into off-the-shelf embedding models without requiring specialized architectures.
Methods
The core contribution is a novel data augmentation technique. Instead of building complex multi-view models, the authors propose treating discrete metadata features (like usernames and locations) as "words" within the existing text embedding framework. To prevent issues with word order in non-textual data, they introduce a padding strategy. Metadata tokens are separated by "dummy" tokens, ensuring that metadata features never appear within the same prediction window as other metadata features. This allows existing text embedding models, specifically paragraph2vec (also known as doc2vec), to directly ingest these augmented data streams. The model then learns a single vector representation for each user that captures information from their tweets, followed accounts, and those accounts' locations.
Datasets
The authors introduce a new dataset collected from Twitter. They sampled followers of 25 US universities, resulting in 5,019 users after filtering. For each user, they collected their 200 most recent tweets and the usernames and locations of up to 100 of their followees. This dataset is used for training and evaluating friend recommendation.
Key Insights and Ideas
- Metadata as "Words": The idea of treating discrete metadata like usernames and locations as if they were words in a text corpus is a clever way to leverage existing, powerful text embedding models.
- Dummy Token Padding: The padding strategy is crucial. It elegantly solves the problem of word order sensitivity in text models when applied to unordered metadata, preventing spurious correlations.
- Friend Locations Matter: Introducing the locations of a user's friends (rather than the user's own location) proved to be a surprisingly informative feature for predicting relationships. This highlights that indirect information can be more powerful than direct attributes.
- Joint Modeling is Superior: Combining different feature types within a single model (jointly) significantly outperforms training separate models for each feature type and then combining their outputs. This suggests that there are complex, informative interactions between text content and metadata that are lost in independent modeling.
Key Results
The proposed method, using paragraph2vec with augmented text, usernames, and friend locations, significantly outperformed baselines.
- Precision and Recall Gains: The full model achieved substantial improvements in precision and recall compared to a text-only baseline, with p-values < 0.001.
- Outperforming Baselines: The joint model consistently outperformed text-only embeddings, text + username embeddings, and even TF-IDF representations in many cases, particularly in precision and mean reciprocal rank (MRR). For example, the joint model achieved a precision of 0.193 at k=10, compared to 0.133 for text-only and 0.174 for TF-IDF.
- Padding Effectiveness: While metadata improved performance even without padding, the padding strategy led to even better results.
Limitations and Future Work
- Dataset Size: The authors acknowledge that the relatively small dataset size might explain why TF-IDF performed competitively with some of their embedding models. Larger datasets could further highlight the advantages of learned embeddings.
- Generalizability: While the method is presented as generalizable to other embedding models and feature types, the paper focuses specifically on
paragraph2vecand Twitter data. Future work could explore its application to different models (e.g., Word2Vec, GloVe) and other social media platforms or data sources. - Feature Engineering: The paper focuses on a specific set of metadata (usernames, friend locations). Exploring other metadata features (e.g., user bio, engagement metrics) could further enhance user representations.
Summary updated