A Markovian approach to distributional semantics with application to semantic compositionality

Edouard Grave, Guillaume Obozinski, Francis R. BachCOLING 2014 · OpenReview

Abstract

Édouard Grave, Guillaume Obozinski, Francis Bach. Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers.

2014.

Introduction

Distributional semantics is founded on the hypothesis that words appearing in similar contexts share similar meanings. While traditional models represent words as static vectors in high-dimensional space, they struggle with two fundamental linguistic phenomena: contextual disambiguation and semantic compositionality. Static vectors often fail to capture how a word's meaning shifts in different contexts (e.g., "head" as a body part vs. "head" as a leader) and struggle to model how individual word meanings combine into complex phrases without ignoring syntactic structure.

The Generative Model

To address these challenges, the authors propose a probabilistic, generative framework that models sentences as Hidden Markov Models (HMMs) structured over syntactic dependency trees. Unlike standard linear HMMs, this model uses the "skeleton" of the sentence—its dependency structure—to capture interactions between words.

Dependency tree structure and corresponding graphical model.

In this model, each word is associated with a latent semantic class. The generation process starts at the root of the dependency tree, where the semantic class of a child node is determined by its parent's class (transition probability), and the specific word is then chosen based on that class (emission probability). This allows the model to leverage syntactic neighbors (like a verb and its object) as the primary influence on a word's meaning.

Representations and Disambiguation

The model produces two types of word representations:

  1. In-context: A vector of posterior probabilities over latent classes, calculated using a message-passing algorithm for a specific sentence.
  2. Out-of-context: A general representation formed by averaging in-context vectors across a large corpus (ukWac).

A key insight of this approach is its ability to perform probabilistic disambiguation. As shown in the visualization below, the out-of-context representation for a polysemous word like "head" resides between distinct semantic clusters.

Visualization of word sense disambiguation for the word "head".

When the word is placed in a specific context, the posterior distribution shifts toward the relevant cluster (e.g., toward "body parts" or "leaders"), effectively resolving the ambiguity through the influence of syntactic neighbors.

Compositionality and Selectional Preferences

Composition is handled by either using the in-context representation of the phrase's "head" word or by summing the representations of all words in the phrase. To compare different parts of speech (e.g., an adjective and a noun), the model utilizes the transition matrix to project representations into the same semantic space. This mechanism allows the model to capture selectional preferences and specific semantic relations like co-hyponymy and meronymy more effectively than flat-window models.

Key Results

The model was evaluated on several benchmarks, demonstrating significant improvements over traditional vector-based methods:

  • Compositionality: On Subject-Verb-Object (SVO) triples, the model achieved a Spearman correlation of 0.41, outperforming the previous state-of-the-art tensor-based approach (0.37).
  • Similarity vs. Relatedness: The model excels at capturing taxonomic similarity (synonyms) rather than general topical relatedness, due to its focus on functional syntactic roles.
  • Metric Selection: The authors found that probabilistic distance metrics, such as Hellinger distance and Jensen-Shannon divergence, are more effective than standard Cosine similarity for these representations.

Conclusion

By integrating syntactic structure into a generative probabilistic framework, this research provides a robust method for handling word sense disambiguation and phrase composition. While the current model ignores specific dependency labels, it establishes a powerful foundation for future work in structured distributional semantics.

Summary updated

BibTeX
@inproceedings{grave2014markovian,
  author = {Edouard Grave and Guillaume Obozinski and Francis R. Bach},
  title = {{A Markovian approach to distributional semantics with application to semantic compositionality}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2014},
  url = {https://openreview.net/forum?id=Far5h4ijxkC},
}
Figure