Abstract
Traditional social media filtering often relies on popularity or recency, failing to account for the author's domain-specific knowledge. This paper introduces a framework for modeling proficiency—a graded level of skill—using unsupervised user embeddings. By treating a user's post history as a single document, the researchers develop "User2Vec" (U2V) representations that capture implicit expertise. Our evaluation across Reddit and Twitter datasets demonstrates that embedding-based models significantly outperform traditional frequency-based methods, particularly in identifying professions with diverse vocabularies like journalism.
Introduction
The digital landscape is saturated with content, yet algorithms rarely distinguish between casual observers and knowledgeable contributors. While "expert identification" focuses on individuals with formal credentials, the vast majority of valuable social media content comes from "proficient" users—hobbyists and enthusiasts who possess significant domain familiarity. This paper argues for proficiency modeling as a graded metric to improve content ranking and information reliability in online communities.
Methodology
The study frames proficiency modeling as an unsupervised learning task, assuming that a user's linguistic patterns correlate with their domain exposure. Five modeling approaches were compared:
- Term Frequency (TF): A baseline measuring specific query word frequency.
- TF-IDF: Normalizes TF by population-wide word frequency.
- User2Vec (U2V): The primary innovation, which generates 400-dimensional embeddings from a user's entire post history. Proficiency is calculated based on the semantic distance between user and word embeddings.
- Relative User2Vec (Rel-U2V): An extension of U2V that normalizes scores against population averages to penalize common terms.
- Latent Dirichlet Allocation (LDA): A standard generative model for topic discovery.
These features were evaluated using a Support Vector Machine (SVM) to classify users into proficiency categories.
Datasets
Two real-world datasets provided the basis for evaluation:
- MHP (Mental Health Professionals): Reddit data distinguishing between clinicians and peers.
- CELEBRITIES: Twitter data where verified occupations (e.g., politics, sports, journalism) served as proxies for proficiency.
Key Insights: Semantic Breadth and Clustering
A major advantage of the User2Vec approach is its ability to capture semantic breadth. Unlike frequency models requiring exact keyword matches, U2V identifies proficiency through related terms and synonyms. This robustness is particularly evident in the model's ability to naturally cluster users.
As shown in the PCA visualization above, the model successfully differentiates between sub-groups (e.g., American football vs. association football players) based purely on linguistic fingerprints, without explicit category labels.
Results and Performance
The embedding-based models (U2V and Rel-U2V) demonstrated clear superiority on the complex CELEBRITIES dataset.
In the multilabeling scenario, U2V achieved an F1 score of 0.82, significantly higher than the 0.75 achieved by TF. Notably, Rel-U2V solved the "journalism gap" where traditional models failed (F1 of 0.62 vs. 0.01 for TF), capturing the subtle professional vocabulary of journalists who often write across varied topics.
Discussion and Future Work
While effective, the current User2Vec approach faces limitations, such as the loss of syntactic nuance due to its "bag-of-words" nature. Future research will explore transformer-based models to capture word order and context. Additionally, the authors aim to refine the model to account for "brevity bias" and to develop post-level proficiency benchmarks, moving beyond user-level aggregation to evaluate individual contributions.
Summary updated