A Method of Short Text Representation Based on the Feature Probability Embedded Vector

Wanting Zhou, Hanbin Wang, Hongguang Sun, Tieli Sun2019 · DOI

TL;DR

This paper introduces a novel unsupervised method for representing short texts by combining traditional feature weighting (TF-IDF), word embeddings (Word2Vec), and topic modeling (LDA). The proposed FPW (Feature, Probability, and Word Embedding) model aims to overcome the limitations of traditional Bag-of-Words (BoW) models, such as high dimensionality and sparsity, by creating more informative and compact text representations.

What Problem Does This Paper Address?

Natural Language Processing (NLP) heavily relies on effectively representing text data for various tasks like classification and sentiment analysis. Traditional methods, such as the Bag-of-Words (BoW) model, treat documents as simple collections of words, leading to high-dimensional and sparse representations that ignore word semantics and context. While deep learning methods have shown promise, this paper seeks to bridge the gap by proposing an unsupervised approach that combines the strengths of shallow methods (feature weighting, topic modeling) with the power of word embeddings. The goal is to create robust text representations that are both informative and computationally efficient, especially for short texts where traditional methods might struggle.

Methods

The paper proposes several models, with the core contribution being the Feature, Probability, and Word Embedding (FPW) model. Here's a breakdown of the key components:

  • Word Embedding (Word2Vec): This technique is used to convert individual words into dense, multi-dimensional vectors. This captures semantic relationships between words, addressing a key limitation of BoW. The paper utilizes the Skip-Gram model from Word2Vec.
  • Feature Weighting (TF-IDF): The traditional Term Frequency-Inverse Document Frequency (TF-IDF) method is employed to assign weights to words based on their importance within a document and across the corpus. This helps in identifying key terms.
  • Topic Modeling (LDA): Latent Dirichlet Allocation (LDA) is used to discover underlying topics within the corpus. This approach represents documents as probability distributions over these topics, providing a different perspective on document content than simple word counts.
  • Proposed Models:
    • FW (Feature and Word Embedding): Combines TF-IDF weighted features with Word2Vec embeddings.
  • PW (Probability and Word Embedding): Combines LDA topic probabilities with Word2Vec embeddings.
  • FPW (Feature, Probability, and Word Embedding): This is the main proposed model, which combines the FW and PW models, aiming to leverage the strengths of all three components. The paper also introduces other variations like FP2 and FPC.

The core idea is to represent documents as vectors derived from these combined techniques, aiming for lower dimensionality and richer semantic information compared to traditional methods.

Datasets

The authors evaluated their models on several publicly available datasets:

  • Amazon_6: A corpus for sentiment analysis tasks, originally containing six categories. Experiments were conducted on balanced subsets of "camera" and "mobile phone" categories, as well as on unbalanced subsets of four classes.
  • FudanNLP: A Chinese text classification corpus with 20 categories. Experiments were performed on balanced subsets of "computer," "economy," and "sports" categories, and also on unbalanced subsets of 17 classes.
  • ChnSentiCorp: A sentiment analysis corpus from Beijing Institute of Technology, containing data for "laptops" and "books," marked with positive and negative emotions.

These datasets were chosen to represent both general classification and sentiment analysis tasks, and to test the models on both balanced and unbalanced data distributions.

Key Insights and Ideas

  • Combining Strengths: The paper strongly advocates for combining different text representation techniques (feature weighting, word embeddings, topic models) to achieve superior performance. No single method consistently outperforms others across all datasets and classifiers.
  • Dimensionality Reduction: The proposed models, particularly FPW, achieve competitive or better results than baselines while using significantly lower dimensional vectors compared to traditional sparse representations.
  • Unsupervised Approach: The models are unsupervised, meaning they don't require labeled data for training the representation itself, making them applicable in scenarios with limited labeled data.
  • Stability: The combination of methods, especially FPW and FPC, demonstrates greater stability across different datasets and classifiers compared to individual baseline methods.
  • Word Embeddings are Crucial: The inclusion of word embeddings (Word2Vec) consistently improves performance, especially when combined with other techniques.
  • Topic Models Offer Complementary Information: LDA, while a shallow method, provides a different perspective on document content by capturing topical themes, which complements feature-based approaches.

Key Results

The proposed FPW model generally achieved competitive or superior results compared to baseline methods (TF-IDF, LDA, Word2Vec) across various datasets and classifiers (KNN, SVM).

  • Amazon_6 Corpus: FPW achieved high accuracy and F1-scores, often outperforming baselines, especially in SVM experiments. For instance, in Table 2 (SVM, 4000 texts), FPW achieved an accuracy of 0.9743 and F1 of 0.9737 in 500 dimensions.
  • FudanNLP Corpus: FPW and FPC models consistently showed improvement over baselines, particularly on unbalanced data. In Table 5 (KNN, 1500 texts), FPW achieved an accuracy of 0.96 and F1 of 0.9598 in 300 dimensions.
  • ChnSentiCorp Corpus: For the "laptops" domain, FPW and FPC performed well with SVM. In the "books" domain, FPW and FPC also showed strong results with KNN, with FPW achieving an accuracy of 0.9019 in 300 dimensions.
  • Dimensionality: The experiments suggest that lower dimensions (e.g., 200-300) often yield optimal or near-optimal results, indicating that high dimensionality is not always necessary for effective representation.

Limitations and Future Work

  • Vector Multiplication Issues: The paper notes that the FW model's performance can sometimes be worse than TF-IDF, possibly due to the algebraic sign changes introduced by vector multiplication, which might alter the meaning of certain values. The authors suggest that improvements in symbolic changes during normalization could enhance effectiveness.
  • P2 Model Limitations: The P2 model, which truncates LDA's topic-word matrix, showed poorer performance, indicating that discarding information can be detrimental. However, combining it with FW (FP2) showed improvement.
  • Classifier Dependence: The performance of models can vary significantly depending on the chosen classifier and corpus characteristics. For instance, SVM struggled with the P2 model on certain corpora.
  • Future Work: The authors plan to explore new deep learning models and different feature weighting techniques to further improve text representation. They also aim to investigate more interesting cooperation modes between different modeling strategies.

Summary updated

BibTeX
@misc{zhou2019method,
  author = {Wanting Zhou and Hanbin Wang and Hongguang Sun and Tieli Sun},
  title = {{A Method of Short Text Representation Based on the Feature Probability Embedded Vector}},
  year = {2019},
  doi = {10.3390/s19173728},
}
Figure