Abstract: We present a novel compositional, generative model for vector space representations of meaning. This model reformulates earlier tensor-based approaches to vector space semantics as a top-down process, and provides efficient algorithms for transformation from natural language to vectors and from vectors to natural language. We describe procedures for estimating the parameters of the model from positive examples of similar phrases, and from distributional representations, then use these procedures to obtain similarity judgments for a set of adjective-noun pairs. The model’s estimation of the similarity of these pairs correlates well with human annotations, demonstrating a substantial improvement over several existing compositional approaches in both settings.
Introduction
In Natural Language Processing (NLP), representing individual words as high-dimensional vectors has become a standard practice. However, scaling these representations to larger structures like phrases and sentences remains a significant challenge. Traditional "bottom-up" compositional models typically combine word vectors through mathematical operations (like addition or matrix multiplication) to derive a phrase vector. These methods often struggle with data sparsity and lack interpretability, as they rarely offer a mechanism to generate natural language back from a point in semantic space.
This paper introduces a generative, top-down framework that treats language as a decoding process. Instead of building meaning from words, we model how a latent concept vector generates a phrase. This approach provides a unified probabilistic framework for both analysis (text-to-vector) and generation (vector-to-text).
Methodology: The Generative Framework
The core of our proposal is a generative process for phrases, specifically adjective-noun pairs. As illustrated in the graphical model below, the process begins with a latent concept vector $x$.
In this model, the adjective $a$ is first chosen based on $x$. An adjective-specific matrix $E_a$ is then selected and applied to $x$ (with added Gaussian noise) to produce a new vector $y$, which represents the noun's semantic content. Finally, the noun $n$ is generated based on $y$.
Crucially, this reverses the standard compositional logic. In bottom-up models, an adjective matrix typically adds information to a noun. In our top-down model, the matrix $E_a$ acts as an information filter, removing the adjective's specific semantic constraints from the phrase vector to recover the underlying noun. Furthermore, mapping a phrase back to its latent vector $x$ (analysis) is formulated as a convex optimization problem, ensuring efficient and reliable computation.
Experimental Evaluation
To evaluate the model, we utilized three primary data sources:
- Wikipedia: A corpus of 10 million tokens used to estimate initial distributional vectors for nouns.
- Matching Pairs: A novel training set of 1,104 adjective-noun pairs extracted from parallel human translations (e.g., "great success" and "huge victory"), providing examples of synonymous phrases.
- Mitchell and Lapata (2010): A benchmark dataset of 396 adjective-noun pairs with human similarity scores used for final testing.
Key Results
The generative model (GEN) demonstrated a significant improvement over existing compositional baselines. When trained on matching pairs, the model achieved a Spearman’s correlation ($ ho$) of 0.365 with human similarity judgments, nearly tripling the performance of the Adjective-Specific Linear Map (ALM) baseline in the same setting.
As shown in the table above, the GEN model significantly outperforms simple additive (+) and noun-only baselines. Interestingly, the model's performance is highest when trained on "Matching pairs"—synonymous phrases—suggesting that learning from human-aligned semantic equivalents is more effective than relying solely on raw distributional data. Qualitatively, the model also succeeded in generation tasks, correctly identifying "fundamental principles" and "main principles" as realizations for a vector representing "basic principles."
Conclusion and Future Directions
By framing semantics as a generative process, we have developed a bidirectional model that excels at capturing human-like similarity judgments. While this study focused on adjective-noun pairs, the framework is designed to integrate with Combinatory Categorial Grammar (CCG) to handle more complex sentence structures. Future work will focus on scaling the training data and allowing noun vectors to be learned as free parameters to further enhance semantic accuracy.
Summary updated