Introduction
Modern word embedding techniques largely rely on the distributional hypothesis, which posits that words appearing in similar contexts share similar semantic meanings. While this approach has revolutionized natural language processing, it faces significant challenges in sentiment analysis and topic classification. For instance, words like "good" and "bad" often appear in identical local contexts (e.g., "This is a [word] movie"), causing standard models to embed them closely and complicating the task for downstream classifiers.
Furthermore, popular models like Paragraph Vector (PV-DBOW) typically focus on local sliding windows of 5-10 words. This narrow focus often fails to capture long-range dependencies or broader "topical" meanings that span an entire document. Simple word co-occurrence counts are often insufficient, as many frequent pairings are merely "noisy coincidences" rather than meaningful semantic units.
Methodology: Global PV-DBOW and TPD
To address these limitations, this research introduces Global PV-DBOW (Glo-PV-DBOW). This model extends the standard objective function by integrating three distinct components: local context (sliding window), document indicators (paragraph vectors), and a novel Global Context component based on Theta Pure Dependence (TPD).
Theta Pure Dependence
Derived from Information Geometry, TPD identifies "pure" dependency patterns. A pattern is considered pure if its joint probability distribution cannot be factorized into simpler sub-patterns. For example, while "London, Chelsea, Sherlock Holmes" might co-occur, "Chelsea" and "Sherlock Holmes" are independent topics linked only by the location "London." Conversely, a pattern like "Climate, conference, Copenhagen" represents a single, inseparable high-level semantic entity (referring to the 2009 UN Climate Change Conference).
By using a Log-Likelihood Ratio Test to identify these pure patterns across a corpus, the authors train "dependence vectors" alongside traditional word and document vectors, allowing the model to capture dependencies between words that may never appear in the same local window but are semantically inseparable within a document's topic.
Experimental Evaluation
The model was evaluated on two primary benchmarks: the IMDB Movie Reviews dataset (50,000 reviews for binary sentiment classification) and the 20 Newsgroups collection. For the latter, the authors focused on binary sub-problems, such as distinguishing between comp.sys.ibm.pc.hardware and comp.sys.mac.hardware, to evaluate topic discovery capabilities.
Key Results and Discussion
Experimental results demonstrate that Glo-PV-DBOW-TPD significantly outperforms standard baselines. On the IMDB sentiment analysis task, the model achieved a state-of-the-art accuracy of 95.05%, surpassing the original Paragraph Vector (92.58%) and other strong models like NBSVM-bi.
In topic discovery tasks using the 20 Newsgroups "comp" sub-dataset, the model achieved 95.47% accuracy, a substantial improvement over the 92.60% achieved by standard PV-DBOW. The research also highlighted that high-order dependencies (e.g., 5-word patterns) provide richer global context than simpler 2-order dependencies.
Qualitative Analysis
Beyond quantitative metrics, qualitative analysis reveals the model's superior ability to distinguish between related but distinct concepts. While standard models might group "IBM" and "Mac" together under a generic "computer" umbrella, Glo-PV-DBOW successfully separates them by recognizing their distinct "pure" global associations, such as linking "Mac" specifically with "Apple" and "Quicktime."
Conclusion
By reinforcing word embeddings with "pure" global context, Glo-PV-DBOW effectively bridges the gap between local distributional semantics and high-level topical dependencies. While the extraction of TPD patterns introduces some preprocessing overhead, the resulting gains in classification accuracy and semantic clarity suggest that "dependence distributed representations" hold significant promise for future applications in information retrieval and text summarization.
Summary updated