Abstract
Existing phrase embedding models struggle with "hybrid" phrases—those that are partly idiomatic and partly literal (e.g., "New York City"). This paper introduces the Hierarchical Compositional (HC) model, which adaptively learns whether to treat a phrase as a single unit, a sum of its parts, or a mixture of both. By utilizing a "composability" parameter and a joint learning process, the HC model significantly improves performance on semantic similarity and analogy tasks.
Introduction
In Natural Language Processing (NLP), representing phrases as vectors (embeddings) is a foundational task. Traditionally, researchers have used two opposing strategies:
- Compositional Models: Based on Frege’s Principle, these assume a phrase's meaning is the sum of its parts (e.g., "North America" = North + America). This fails for idioms like "hot dog."
- Non-Compositional Models: These treat every phrase as a unique, atomic unit (e.g., "New York"). This ignores internal structure and suffers from data sparsity—requiring many examples of the exact phrase to learn a good vector.
The authors identify a missing middle ground: hybrid-compositionality. A phrase like "New York City" is a mix; "New York" is a non-compositional unit, but "City" is a literal addition to it. Existing models force a choice between "all or nothing," which limits their accuracy for long, complex phrases.
Methodology
The proposed Hierarchical Compositional Phrase Embedding (HCPE) framework introduces a "composability" parameter, $\beta(p)$, which acts as a weighted dial between literal and idiomatic interpretations.
The Model
A phrase's vector is defined as a mixture: $vec(p) = (1 - \beta(p)) \cdot e(p) + \beta(p) \cdot \sum vec(constituents)$ Where $e(p)$ is the unique "idiomatic" vector and the sum represents the "literal" composition.
Joint Learning (Hard-EM)
The model learns both vectors and the best way to "partition" a phrase through a two-step iterative process:
- E-step: Uses Dynamic Programming to find the most likely hierarchical structure (e.g., deciding that "New York City" should be split into ["New York", "City"] rather than ["New", "York City"]).
- M-step: Updates the word/phrase vectors and the $\beta$ parameters using Stochastic Gradient Descent (SGD).
The authors observed that $\beta$ tends to follow a "U-shaped" distribution, meaning phrases are usually either very literal or very idiomatic. A specialized learning rate was implemented to speed up training when $\beta$ is uncertain (near 0.5).
Experimental Evaluation
The model was trained on massive corpora, including Wikipedia (2 billion words) and Google News (0.28 billion words), and evaluated on standard phrase similarity and analogy benchmarks.
Key Results
HCPE consistently outperformed standard Word2vec and the SVO model. On the Google News corpus, HCPE achieved a 5.5% improvement in precision for analogy tasks and a 1.8% average improvement across phrase similarity benchmarks.
As shown in the figures above, HCPE reaches high correlation scores with significantly less training data than Word2vec, suggesting its hierarchical structure helps it learn more "meaning" per word.
In analogy tasks, the model's ability to fine-tune component word embeddings based on their contribution to phrases leads to superior precision, as illustrated in the performance trends across different data scales.
Conclusion
The HCPE framework successfully bridges the gap between compositional and non-compositional models by adaptively learning phrase structures. While computationally more intensive than simpler models, its ability to capture hybrid meanings leads to significant gains in semantic representation. Future work may explore more sophisticated composition functions, such as Recurrent Neural Networks, to further enhance the model's hierarchical capabilities.
Summary updated