Increasing In-Class Similarity by Retrofitting Embeddings with Demographic Information

Dirk Hovy, Tommaso Fornaciari2018 · DOI

Abstract

Most text-classification approaches represent the input based on textual features, either feature-based or continuous. However, this ignores strong non-linguistic similarities like homophily: people within a demographic group use language more similar to each other than to non-group members. We use homophily cues to retrofit text-based author representations with non-linguistic information, and introduce a trade-off parameter. This approach increases in-class similarity between authors, and improves classification performance by making classes more linearly separable. We evaluate the effect of our method on two author-attribute prediction tasks with various training set sizes and parameter settings. We find that our method can significantly improve classification performance, especially when the number of labels is large and limited labeled data is available.

Introduction

In Natural Language Processing (NLP), predicting author attributes such as age or gender typically relies on linguistic features—either traditional n-grams or continuous embeddings like Doc2Vec. However, social media text is often too short or noisy for these features to capture a clear signal.

This research addresses the omission of a fundamental sociological principle: homophily. People within a demographic group share stylistic and topical similarities that make their language more similar to each other than to outsiders. While this similarity is known, standard embedding techniques do not explicitly group authors of the same class together in vector space, complicating the task for classifiers to draw distinct boundaries between categories.

Methodology: Retrofitting for Homophily

The researchers adapt a technique called retrofitting, originally designed to refine word vectors using external lexicons. The process involves three main steps:

  1. Initial Embeddings: Authors are represented using Doc2Vec (paragraph vectors) based on their text corpus.
  2. Retrofitting Step: For training data, each author's vector is iteratively updated as a weighted combination of its original linguistic representation and the average vector of all other authors in the same demographic group. A trade-off parameter, $\alpha$, is introduced: a low $\alpha$ prioritizes the group's "average" identity (homophily), while a high $\alpha$ preserves the individual's unique linguistic output.
  3. Transformation Matrix ($T$): To handle unlabeled test data, a linear transformation matrix ($T$) is learned to map the original linguistic space to the new, retrofitted space. This allows for the projection of unknown authors into the optimized space before classification.

Finally, a simple Logistic Regression classifier is trained on these optimized representations.

Key Insights: Linear Separability and Group Signal

The core advantage of retrofitting is its role as a supervised dimensionality refinement tool. By explicitly pulling same-class authors together, the classes become more linearly separable.

Retrofitted author embeddings showing increased class clustering and improved linear separability.

The study found that the best results were achieved with a low $\alpha$ (e.g., 0.01), suggesting that for demographic prediction, shared group traits are often a more reliable signal than the nuances of an individual's short text. Furthermore, the method is particularly effective for complex tasks with many classes (like age prediction) and when training data is limited, effectively "amplifying" the signal in small datasets.

Experimental Results

The method was evaluated using a large-scale dataset of online reviews (98,608 authors, 8 million words) for age and gender prediction.

Classification performance (F1) for age (top) and gender (bottom) across various training sizes and alpha values. Retrofitting (red) consistently outperforms standard embeddings (dashed) and BOW (dotted) for age prediction.
  • Age Prediction (10 classes): Retrofitting significantly outperformed both standard Doc2Vec embeddings and Bag-of-Words (BOW) models. With $\alpha = 0.01$, the Micro-F1 score was consistently higher across all training set sizes (1k to 10k instances), with highly significant p-values ($p < 0.0001$).
  • Gender Prediction (2 classes): While retrofitting showed a slight upward trend, the results were not statistically significant compared to standard embeddings. This suggests that binary classification is a simpler task where the linguistic signal is already relatively clear.

Limitations and Future Directions

The method currently relies on a labeled training set to build the transformation matrix $T$, meaning noisy or biased initial labels could be amplified. Additionally, the linear assumption of the transformation matrix $T$ might be limiting; future work could explore non-linear transformations like neural networks to capture more complex relationships. Despite these limitations, the authors propose retrofitting as a general-purpose preprocessing step applicable to various text-classification tasks where class-based similarity is expected.

Summary updated

Abstract

Most text-classification approaches represent the input based on textual features, either feature-based or continuous. However, this ignores strong non-linguistic similarities like homophily: people within a demographic group use language more similar to each other than to non-group members. We use homophily cues to retrofit text-based author representations with non-linguistic information, and introduce a trade-off parameter. This approach increases in-class similarity between authors, and improves classification performance by making classes more linearly separable. We evaluate the effect of our method on two authorattribute prediction tasks with various trainingset sizes and parameter settings. We find that our method can significantly improve classification performance, especially when the number of labels is large and limited labeled data is available. It is potentially applicable as preprocessing step to any text-classification task.
BibTeX
@misc{hovy2018increasing,
  author = {Dirk Hovy and Tommaso Fornaciari},
  title = {{Increasing In-Class Similarity by Retrofitting Embeddings with Demographic Information}},
  year = {2018},
  doi = {10.18653/v1/d18-1070},
}
Figure