An Adaptive Hierarchical Compositional Model for Phrase Embedding

Bing Li, Xiaochun Yang, Bin Wang, Wei Wang, Wei Cui, Xianchao Zhang

Abstract

Existing phrase embedding models struggle with "hybrid" phrases—those that are partly idiomatic and partly literal (e.g., "New York City"). This paper introduces the Hierarchical Compositional (HC) model, which adaptively learns whether to treat a phrase as a single unit, a sum of its parts, or a mixture of both. By utilizing a "composability" parameter and a joint learning process, the HC model significantly improves performance on semantic similarity and analogy tasks.

Introduction

In Natural Language Processing (NLP), representing phrases as vectors (embeddings) is a foundational task. Traditionally, researchers have used two opposing strategies:

  1. Compositional Models: Based on Frege’s Principle, these assume a phrase's meaning is the sum of its parts (e.g., "North America" = North + America). This fails for idioms like "hot dog."
  2. Non-Compositional Models: These treat every phrase as a unique, atomic unit (e.g., "New York"). This ignores internal structure and suffers from data sparsity—requiring many examples of the exact phrase to learn a good vector.

The authors identify a missing middle ground: hybrid-compositionality. A phrase like "New York City" is a mix; "New York" is a non-compositional unit, but "City" is a literal addition to it. Existing models force a choice between "all or nothing," which limits their accuracy for long, complex phrases.

Methodology

The proposed Hierarchical Compositional Phrase Embedding (HCPE) framework introduces a "composability" parameter, $\beta(p)$, which acts as a weighted dial between literal and idiomatic interpretations.

The Model

A phrase's vector is defined as a mixture: $vec(p) = (1 - \beta(p)) \cdot e(p) + \beta(p) \cdot \sum vec(constituents)$ Where $e(p)$ is the unique "idiomatic" vector and the sum represents the "literal" composition.

Joint Learning (Hard-EM)

The model learns both vectors and the best way to "partition" a phrase through a two-step iterative process:

  • E-step: Uses Dynamic Programming to find the most likely hierarchical structure (e.g., deciding that "New York City" should be split into ["New York", "City"] rather than ["New", "York City"]).
  • M-step: Updates the word/phrase vectors and the $\beta$ parameters using Stochastic Gradient Descent (SGD).

The authors observed that $\beta$ tends to follow a "U-shaped" distribution, meaning phrases are usually either very literal or very idiomatic. A specialized learning rate was implemented to speed up training when $\beta$ is uncertain (near 0.5).

Experimental Evaluation

The model was trained on massive corpora, including Wikipedia (2 billion words) and Google News (0.28 billion words), and evaluated on standard phrase similarity and analogy benchmarks.

Key Results

HCPE consistently outperformed standard Word2vec and the SVO model. On the Google News corpus, HCPE achieved a 5.5% improvement in precision for analogy tasks and a 1.8% average improvement across phrase similarity benchmarks.

Spearman correlation results for varying vector dimensions and data sizes. HCPE shows consistent superiority and better data efficiency compared to baselines.

As shown in the figures above, HCPE reaches high correlation scores with significantly less training data than Word2vec, suggesting its hierarchical structure helps it learn more "meaning" per word.

Precision at top-1 (%) for analogy tasks across varying data sizes and vector dimensions. HCPE maintains a performance lead over Word2vec and SVO.

In analogy tasks, the model's ability to fine-tune component word embeddings based on their contribution to phrases leads to superior precision, as illustrated in the performance trends across different data scales.

Conclusion

The HCPE framework successfully bridges the gap between compositional and non-compositional models by adaptively learning phrase structures. While computationally more intensive than simpler models, its ability to capture hybrid meanings leads to significant gains in semantic representation. Future work may explore more sophisticated composition functions, such as Recurrent Neural Networks, to further enhance the model's hierarchical capabilities.

Summary updated

Abstract

Phrase embedding aims at representing phrases in a vector space and it is important for the performance of many NLP tasks. Existing models only regard a phrase as either full-compositional or non-compositional, while ignoring the hybrid-compositionality that widely exists, especially in long phrases. This drawback prevents them from having a deeper insight into the semantic structure for long phrases and as a consequence, weakens the accuracy of the embeddings. In this paper, we present a novel method for jointly learning compositionality and phrase embedding by adaptively weighting different compositions using an implicit hierarchical structure. Our model has the ability of adaptively adjusting among different compositions without entailing too much model complexity and time cost. To the best of our knowledge, our work is the first effort that considers hybrid-compositionality in phrase embedding. The experimental evaluation demonstrates that our model outperforms state-of-the-art methods in both similarity tasks and analogy tasks.
BibTeX
@inproceedings{li2018adaptive,
  author = {Bing Li and Xiaochun Yang and Bin Wang and Wei Wang and Wei Cui and Xianchao Zhang},
  title = {{An Adaptive Hierarchical Compositional Model for Phrase Embedding}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/576},
}
Figure