Modeling Proficiency with Implicit User Representations

Kim Breitwieser, Allison Lahnala, Charles Welch, Lucie Flek, Martin PotthastarXiv 2021 · arXiv

Abstract

Traditional social media filtering often relies on popularity or recency, failing to account for the author's domain-specific knowledge. This paper introduces a framework for modeling proficiency—a graded level of skill—using unsupervised user embeddings. By treating a user's post history as a single document, the researchers develop "User2Vec" (U2V) representations that capture implicit expertise. Our evaluation across Reddit and Twitter datasets demonstrates that embedding-based models significantly outperform traditional frequency-based methods, particularly in identifying professions with diverse vocabularies like journalism.

Introduction

The digital landscape is saturated with content, yet algorithms rarely distinguish between casual observers and knowledgeable contributors. While "expert identification" focuses on individuals with formal credentials, the vast majority of valuable social media content comes from "proficient" users—hobbyists and enthusiasts who possess significant domain familiarity. This paper argues for proficiency modeling as a graded metric to improve content ranking and information reliability in online communities.

Methodology

The study frames proficiency modeling as an unsupervised learning task, assuming that a user's linguistic patterns correlate with their domain exposure. Five modeling approaches were compared:

  1. Term Frequency (TF): A baseline measuring specific query word frequency.
  2. TF-IDF: Normalizes TF by population-wide word frequency.
  3. User2Vec (U2V): The primary innovation, which generates 400-dimensional embeddings from a user's entire post history. Proficiency is calculated based on the semantic distance between user and word embeddings.
  4. Relative User2Vec (Rel-U2V): An extension of U2V that normalizes scores against population averages to penalize common terms.
  5. Latent Dirichlet Allocation (LDA): A standard generative model for topic discovery.

These features were evaluated using a Support Vector Machine (SVM) to classify users into proficiency categories.

Datasets

Two real-world datasets provided the basis for evaluation:

  • MHP (Mental Health Professionals): Reddit data distinguishing between clinicians and peers.
  • CELEBRITIES: Twitter data where verified occupations (e.g., politics, sports, journalism) served as proxies for proficiency.

Key Insights: Semantic Breadth and Clustering

A major advantage of the User2Vec approach is its ability to capture semantic breadth. Unlike frequency models requiring exact keyword matches, U2V identifies proficiency through related terms and synonyms. This robustness is particularly evident in the model's ability to naturally cluster users.

2D PCA representation of User2Vec embeddings showing natural clustering of different types of athletes.

As shown in the PCA visualization above, the model successfully differentiates between sub-groups (e.g., American football vs. association football players) based purely on linguistic fingerprints, without explicit category labels.

Results and Performance

The embedding-based models (U2V and Rel-U2V) demonstrated clear superiority on the complex CELEBRITIES dataset.

Comparison of model performance across different scenarios in the CELEBRITIES dataset.

In the multilabeling scenario, U2V achieved an F1 score of 0.82, significantly higher than the 0.75 achieved by TF. Notably, Rel-U2V solved the "journalism gap" where traditional models failed (F1 of 0.62 vs. 0.01 for TF), capturing the subtle professional vocabulary of journalists who often write across varied topics.

Discussion and Future Work

While effective, the current User2Vec approach faces limitations, such as the loss of syntactic nuance due to its "bag-of-words" nature. Future research will explore transformer-based models to capture word order and context. Additionally, the authors aim to refine the model to account for "brevity bias" and to develop post-level proficiency benchmarks, moving beyond user-level aggregation to evaluate individual contributions.

Summary updated

Abstract

We introduce the problem of proficiency modeling: Given a user's posts on a social media platform, the task is to identify the subset of posts or topics for which the user has some level of proficiency. This enables the filtering and ranking of social media posts on a given topic as per user proficiency. Unlike experts on a given topic, proficient users may not have received formal training and possess years of practical experience, but may be autodidacts, hobbyists, and people with sustained interest, enabling them to make genuine and original contributions to discourse. While predicting whether a user is an expert on a given topic imposes strong constraints on who is a true positive, proficiency modeling implies a graded scoring, relaxing these constraints. Put another way, many active social media users can be assumed to possess, or eventually acquire, some level of proficiency on topics relevant to their community. We tackle proficiency modeling in an unsupervised manner by utilizing user embeddings to model engagement with a given topic, as indicated by a user's preference for authoring related content. We investigate five alternative approaches to model proficiency, ranging from basic ones to an advanced, tailored user modeling approach, applied within two real-world benchmarks for evaluation.
BibTeX
@misc{breitwieser2021modeling,
  author = {Kim Breitwieser and Allison Lahnala and Charles Welch and Lucie Flek and Martin Potthast},
  title = {{Modeling Proficiency with Implicit User Representations}},
  year = {2021},
  eprint = {2110.08011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.08011},
}
Figure