Social Media Text Classification under Negative Covariate Shift

Geli Fei, Bing LiuEMNLP 2015 · DOI

Introduction

In social media analytics, a persistent challenge is the mismatch between training data and real-world deployment. While the "positive" class (e.g., posts about "politics") is often well-defined, the "negative" class—representing everything else—is an infinite, ever-changing sea of topics. This phenomenon is known as negative covariate shift: a classifier trained to distinguish politics from sports and movies may fail when confronted with posts about cooking or technology.

The CBS-L Methodology

To address this, the authors propose Center-Based Similarity Learning (CBS-L). Instead of working in the traditional Document Space (DS) of n-gram frequencies, CBS-L transforms documents into a space defined by their similarity to a positive class "center."

The pipeline involves:

  1. Feature Extraction: Documents are represented using tf-idf weighted unigrams, bigrams, and trigrams.
  2. Finding the Center: The algorithm computes a positive class center vector using the Rocchio method, which subtracts a weighted version of negative documents to "push" the center away from non-discriminative terms.
  3. Space Transformation: Documents are re-represented as vectors of similarity scores (e.g., Cosine, Gower, Lorentzian, Dice, and Jaccard) relative to the center.
  4. Learning: A standard binary SVM is trained on these similarity features.
Conceptual comparison of linear boundaries (solid line) vs. center-based similarity boundaries (dashed circle).

By creating a "similarity bubble," the model effectively rejects any document that is "too different" from the center, regardless of whether that specific negative topic was encountered during training. As illustrated above, a linear boundary might misclassify unseen negative topics (triangles) that fall on the positive side of the hyperplane, whereas the similarity bubble correctly excludes them.

Experimental Evaluation

The method was evaluated using an Amazon review database covering 50 product types. To simulate negative covariate shift, models were trained on a subset of negative topics (varying from 10 to 40) and tested on a "Combined" set containing both seen and unseen negative topics.

Key Results

CBS-L demonstrated significant robustness compared to standard SVMs and One-Class SVMs (OSVM).

Performance metrics (Precision, Recall, F1-score) for in-training, out-of-training, and combined scenarios.

As shown in the results table, when only 10 negative topics were available for training, CBS-L achieved a combined F1-score of 0.481, nearly doubling the performance of a standard SVM (0.262) and a One-Class SVM (0.205). The most dramatic gains occurred in the "Out-of-training" category, where CBS-L maintained much higher precision and recall on topics the model had never encountered. Across all 50 topics, CBS-L reached an average F1-score of 0.598, compared to 0.439 for the baseline SVM.

Conclusion

CBS-L offers a robust solution for social media classification by focusing on the "wholeness" of the positive class rather than just looking for a line of separation. While it requires some negative data for center refinement and assumes a stable positive class, its ability to handle unknown negative topics makes it a powerful tool for real-world deployment where the "irrelevant" data is unpredictable.

Summary updated

BibTeX
@inproceedings{fei2015social,
  author = {Geli Fei and Bing Liu},
  title = {{Social Media Text Classification under Negative Covariate Shift}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1282},
}
Figure