Dataless Short Text Classification Based on Biterm Topic Model and Word Embeddings

Yi Yang, Hongan Wang, Jiaqi Zhu, Yunkun Wu, Kailong Jiang, Wenli Guo, Wandong ShiIJCAI 2020 · DOI

TL;DR

This paper introduces two novel models, SeedBTM and SeedTBTM, for dataless short text classification. By combining biterm topic models with word embeddings, they effectively leverage seed words and semantic relationships to classify short texts, outperforming existing methods, especially on challenging datasets with overlapping categories.

What Problem Does This Paper Address?

The paper tackles the challenge of classifying short texts (like social media posts or news headlines) when labeled data is scarce or non-existent. Traditional supervised methods require extensive labeled datasets, which are expensive and time-consuming to create. Dataless classification aims to overcome this by using only a few "seed words" for each category. However, existing dataless methods often struggle with short texts due to their inherent sparsity and limited word co-occurrence information, which makes it difficult to form robust topics.

Methods

The core innovation lies in a novel approach that integrates two key sources of information:

  • Word Co-occurrence (from Topic Models): The authors build upon Biterm Topic Model (BTM), which models pairs of words (biterms) that co-occur in a text. This helps capture local word relationships.
  • Category-Word Similarity (from Word Embeddings): They leverage pre-trained word embeddings (like GloVe) to calculate the semantic similarity between seed words for a category and all words in the vocabulary. This similarity score acts as a prior knowledge, guiding the topic inference process.

Two specific models are proposed:

  • Seeded Biterm Topic Model (SeedBTM): This model extends the classical BTM by incorporating the category-word similarity scores as a prior. This modification helps the topic model learn category-aware topics, even with limited data.
  • Seeded Twitter Biterm Topic Model (SeedTBTM): This model further extends SeedBTM by adapting it to Twitter-BTM. It incorporates additional user information (like author ID) to enhance classification accuracy, particularly useful for datasets with user-specific patterns.

The inference process uses Gibbs sampling to estimate topic distributions, and document categories are predicted based on the most probable topic.

Datasets

The models were evaluated on five real-world short-text datasets:

  • SearchSnip: A widely used dataset for short text classification, containing web search snippets.
  • Reuters10T: News titles from the Reuters-21578 dataset, focusing on 10 large categories.
  • AGnews: News article titles from the AG News dataset, using 4 large categories.
  • TwitterTrends10: Tweets related to trending events, useful for event classification.
  • News Category Dataset (HuffPost): A large dataset of news headlines, with a focus on sub-datasets (HuffN10, HuffN4-SWPB, HuffN4-ECMC) that contain overlapping and interrelated categories.

The latter two datasets (TwitterTrends10 and the HuffPost sub-datasets) also contain user information, which is leveraged by SeedTBTM.

Key Insights and Ideas

  • Bridging the Gap: The paper successfully bridges the gap between topic modeling and word embeddings for dataless short text classification, showing that combining these two approaches is more powerful than using either alone.
  • Word Embeddings as Priors: Using word embeddings to derive category-word similarity scores as priors for topic models is a flexible and effective way to inject semantic knowledge.
  • Handling Overlapping Categories: The proposed models show particular strength when categories are overlapping and interrelated, a common challenge that simpler methods struggle with. This suggests the models can disentangle nuanced semantic relationships.
  • Metadata Matters: Incorporating user information (as in SeedTBTM) can significantly boost performance, demonstrating the value of leveraging available metadata in dataless scenarios.
  • Domain Adaptability: The models show domain insensitivity, performing well across various datasets, though the choice between external and self-trained word embeddings can impact performance depending on the domain specificity of categories.

Key Results

  • Superior Performance: SeedBTM consistently outperforms state-of-the-art baselines on most datasets, achieving significant improvements in Macro-F1 scores (e.g., 1.2-6.3% increase compared to the best baseline on some datasets).
  • SeedTBTM Advantage: SeedTBTM further improves performance, especially on datasets with user information, showing gains of up to 16.3% in Macro-F1 compared to SeedBTM.
  • Robustness to Seed Word Type: The models achieve comparable performance even with simpler, single-word seed labels (SL) compared to more descriptive seed words (SD), reducing the burden of selecting elaborate seed sets.
  • Parameter Sensitivity: The models are relatively insensitive to parameter choices like iteration number and the similarity threshold (€), indicating good robustness.

Limitations and Future Work

  • Domain-Specific Embeddings: While external word embeddings are generally effective, the paper notes that for domain-specific categories, self-trained word embeddings might yield better results. The authors suggest choosing embeddings based on category scope and granularity.
  • Hierarchical Classification: The authors propose extending their approach to handle hierarchical short text classification in future work.
  • Optimizing Integration: Further research is suggested on optimizing the computation of embedding-based similarity and its integration with topic models.

Summary updated

BibTeX
@inproceedings{yang2020dataless,
  author = {Yi Yang and Hongan Wang and Jiaqi Zhu and Yunkun Wu and Kailong Jiang and Wenli Guo and Wandong Shi},
  title = {{Dataless Short Text Classification Based on Biterm Topic Model and Word Embeddings}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/549},
}
Figure