Reinforcing the Topic of Embeddings with Theta Pure Dependence for Text Classification

Ning Xing, Yuexian Hou, Peng Zhang, Wenjie Li, Dawei Song

Introduction

Modern word embedding techniques largely rely on the distributional hypothesis, which posits that words appearing in similar contexts share similar semantic meanings. While this approach has revolutionized natural language processing, it faces significant challenges in sentiment analysis and topic classification. For instance, words like "good" and "bad" often appear in identical local contexts (e.g., "This is a [word] movie"), causing standard models to embed them closely and complicating the task for downstream classifiers.

Furthermore, popular models like Paragraph Vector (PV-DBOW) typically focus on local sliding windows of 5-10 words. This narrow focus often fails to capture long-range dependencies or broader "topical" meanings that span an entire document. Simple word co-occurrence counts are often insufficient, as many frequent pairings are merely "noisy coincidences" rather than meaningful semantic units.

Methodology: Global PV-DBOW and TPD

To address these limitations, this research introduces Global PV-DBOW (Glo-PV-DBOW). This model extends the standard objective function by integrating three distinct components: local context (sliding window), document indicators (paragraph vectors), and a novel Global Context component based on Theta Pure Dependence (TPD).

Theta Pure Dependence

Derived from Information Geometry, TPD identifies "pure" dependency patterns. A pattern is considered pure if its joint probability distribution cannot be factorized into simpler sub-patterns. For example, while "London, Chelsea, Sherlock Holmes" might co-occur, "Chelsea" and "Sherlock Holmes" are independent topics linked only by the location "London." Conversely, a pattern like "Climate, conference, Copenhagen" represents a single, inseparable high-level semantic entity (referring to the 2009 UN Climate Change Conference).

By using a Log-Likelihood Ratio Test to identify these pure patterns across a corpus, the authors train "dependence vectors" alongside traditional word and document vectors, allowing the model to capture dependencies between words that may never appear in the same local window but are semantically inseparable within a document's topic.

Experimental Evaluation

The model was evaluated on two primary benchmarks: the IMDB Movie Reviews dataset (50,000 reviews for binary sentiment classification) and the 20 Newsgroups collection. For the latter, the authors focused on binary sub-problems, such as distinguishing between comp.sys.ibm.pc.hardware and comp.sys.mac.hardware, to evaluate topic discovery capabilities.

Key Results and Discussion

Experimental results demonstrate that Glo-PV-DBOW-TPD significantly outperforms standard baselines. On the IMDB sentiment analysis task, the model achieved a state-of-the-art accuracy of 95.05%, surpassing the original Paragraph Vector (92.58%) and other strong models like NBSVM-bi.

Table 1: Accuracy rates on the IMDB dataset comparing Glo-PV-DBOW-TPD-5 against various baselines.

In topic discovery tasks using the 20 Newsgroups "comp" sub-dataset, the model achieved 95.47% accuracy, a substantial improvement over the 92.60% achieved by standard PV-DBOW. The research also highlighted that high-order dependencies (e.g., 5-word patterns) provide richer global context than simpler 2-order dependencies.

Qualitative Analysis

Beyond quantitative metrics, qualitative analysis reveals the model's superior ability to distinguish between related but distinct concepts. While standard models might group "IBM" and "Mac" together under a generic "computer" umbrella, Glo-PV-DBOW successfully separates them by recognizing their distinct "pure" global associations, such as linking "Mac" specifically with "Apple" and "Quicktime."

Table 2: Qualitative comparison of word associations between standard PV-DBOW and Glo-PV-DBOW, showing improved brand and component separation.

Conclusion

By reinforcing word embeddings with "pure" global context, Glo-PV-DBOW effectively bridges the gap between local distributional semantics and high-level topical dependencies. While the extraction of TPD patterns introduces some preprocessing overhead, the resulting gains in classification accuracy and semantic clarity suggest that "dependence distributed representations" hold significant promise for future applications in information retrieval and text summarization.

Summary updated

Abstract

For sentiment classification, it is often recognized that embedding based on distributional hypothesis is weak in capturing sentiment contrast‐contrasting words may have similar local context. Based on broader context, we propose to incorporate Theta Pure Dependence (TPD) into the Paragraph Vector method to reinforce topical and sentimental information. TPD has a theoretical guarantee that the word dependency is pure, i.e., the dependence pattern has the integral meaning whose underlying distribution can not be conditionally factorized. Our method outperforms the state-of-the-art performance on text classification tasks.
BibTeX
@inproceedings{xing2015reinforcing,
  author = {Ning Xing and Yuexian Hou and Peng Zhang and Wenjie Li and Dawei Song},
  title = {{Reinforcing the Topic of Embeddings with Theta Pure Dependence for Text Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1305},
}
Figure