Abstract
Traditional text-classification models often struggle with the inherent noise and stylistic variability of social media data. This paper introduces a method to enhance author attribute prediction by leveraging homophily—the tendency for individuals with similar demographic traits to share linguistic patterns. By "retrofitting" linguistic embeddings to pull authors of the same demographic closer together in vector space, the researchers significantly boost classification performance, especially in data-scarce scenarios.
Introduction
Most text-classification models rely purely on linguistic features, such as word choice and stylistic markers. However, in social media contexts, these signals are often weak due to short post lengths and a vast variety of topics, which can obscure the stylistic signatures identifying an author's age or gender.
The authors argue that a powerful non-linguistic signal is being ignored: homophily. If two people belong to the same demographic group, they are statistically more likely to share linguistic patterns than two people from different groups. The challenge lies in integrating this sociological intuition into machine learning models without relying on external dictionaries or massive amounts of labeled data.
Methods and Datasets
The researchers propose a three-step pipeline to transform author representations into more separable clusters:
- Linguistic Representation: Baseline embeddings are created for approximately 98,000 authors using Doc2Vec, representing each author’s collective writings as a 300-dimensional vector.
- Retrofitting: Borrowing a technique used to align word vectors with lexicons, the authors iteratively update training set author vectors. The goal is to move an author's vector closer to the average vector of others in their same class (e.g., the same age group). A trade-off parameter, $\alpha$, balances the original linguistic signal with this new class-similarity signal.
- The Translation Matrix: Since retrofitting requires known labels (unavailable for test data), the authors learn a translation matrix ($T$) using a least-squares fit. This matrix maps the original linguistic space to the retrofitted space, allowing unknown test authors to be transformed before classification.
The study utilizes a large-scale dataset of online reviews (Hovy et al., 2015) containing 98,608 users and roughly 8 million words. Attributes include age (discretized by decade) and gender.
Key Results and Insights
The core finding is that retrofitting significantly improves linear separability. By pulling same-class authors together, the clusters in the vector space become tighter, making it easier for a linear classifier to draw boundaries.
Age and Gender Prediction
- Age Prediction: Retrofitting provided a substantial and statistically significant boost in Micro-F1 scores across all training set sizes. The performance jump was most dramatic at low $\alpha$ values (e.g., $\alpha = 0.01$), particularly when training data was limited (e.g., 1,000 instances).
- Gender Prediction: While retrofitting showed some improvement, the results were not statistically significant. This suggests gender might already be more linearly separable in base features, or gender-based homophily is less pronounced in this dataset.
Data Efficiency and Regularization
The "boost" from retrofitting is most pronounced when training data is scarce, making it a valuable tool for low-resource scenarios. Interestingly, using the approximated transformation (via the translation matrix) sometimes outperformed the "perfect" retrofitted training data, acting as a form of regularization that prevents overfitting.
Discussion and Future Work
While the current method uses a linear translation matrix, the authors acknowledge that non-linear transformations might capture the mapping between spaces even better. Furthermore, the method's success depends on the quality of the initial Doc2Vec embeddings.
The authors believe this approach could be applied to any text-classification task where homophily is expected, such as sentiment, topic, or ideology analysis. Future work could explore applying retrofitting as a standard preprocessing step for a broader range of NLP tasks.
Summary updated