Abstract
Neural language models are a powerful tool to embed words into semantic vector spaces. However, learning such models generally relies on the availability of abundant and diverse training examples. In highly specialised domains this requirement may not be met due to difficulties in obtaining a large corpus, or the limited range of expression in average use. Such domains may encode prior knowledge about entities in a knowledge base or ontology. We propose a generative model which integrates evidence from diverse data sources, enabling the sharing of semantic information. We achieve this by generalising the concept of co-occurrence from distributional semantics to include other relationships between entities or words, which we model as affine transformations on the embedding space. We demonstrate the effectiveness of this approach by outperforming recent models on a link prediction task and demonstrating its ability to profit from partially or fully unobserved data training labels. We further demonstrate the usefulness of learning from different data sources with overlapping vocabularies.
Introduction
Standard neural language models, such as word2vec, typically require massive corpora—often billions of words—to learn high-quality semantic vectors. In specialized domains like medicine, obtaining such large, open-access datasets is challenging due to privacy and availability constraints. Conversely, these fields often possess rich, human-curated structured knowledge in the form of ontologies or knowledge graphs (e.g., "Anemia is a disease").
Historically, models have treated unstructured text and structured relational data as separate information sources. This paper, presented at AAAI 2016, introduces a generative probabilistic model that bridges this gap. By treating every relationship—whether a grammatical co-occurrence in a sentence or a formal ontological link—as an affine transformation in a shared vector space, the model can transfer knowledge between disparate data formats.
Methodology: Unifying Text and Knowledge
The core of the proposed framework is a generative model of triplets $(S, R, T)$, where $S$ is a source word, $R$ is a relationship, and $T$ is a target word.
Relationships as Operators
Unlike models that represent relationships as simple vectors, this approach treats each relationship $R$ as an affine transformation (represented by a matrix $G_R$). To evaluate the likelihood of a triplet, the model transforms the source vector $c_s$ via $G_R$ and measures its cosine similarity to the target vector $v_t$. This "space distortion" allows the model to capture different facets of a word's meaning depending on the relational context.
Source Agnosticism
A key innovation is treating the "appears in a sentence with" relationship from unstructured text identically to formal relationships like "is a hyponym of" from a knowledge base. This allows the model to ingest both unstructured text and structured triples into a single energy function.
Probabilistic Framework
The model employs a Boltzmann distribution. To handle the mathematically intractable partition function over large vocabularies, the authors utilize Persistent Contrastive Divergence (PCD). This technique uses Gibbs sampling to estimate the gradients necessary for training, providing a more rigorous probabilistic interpretation compared to heuristic "negative sampling" methods.
Experimental Results
The model was evaluated using WordNet (a lexical database with ~38,000 words and 11 relationship types) and English Wikipedia.
Link Prediction and Data Synergy
On the WordNet triplet completion task, the model achieved 78.2% accuracy, significantly outperforming the Neural Tensor Model (NTN) and Bilinear models. Crucially, the experiments demonstrated that adding unstructured Wikipedia data consistently improved the prediction of formal relationships in WordNet. This synergy is particularly pronounced when labeled data is scarce.
Latent Relationship Discovery
In unsupervised settings, the model successfully identified "contextual subclasses" in Wikipedia text that mapped to real-world relationships. This resulted in higher-quality embeddings than standard distributional models, as structured knowledge acted as a powerful regularizer, "anchoring" word vectors even with small text corpora.
Conclusion and Future Work
The proposed generative model effectively integrates diverse data sources by generalizing the concept of co-occurrence. While the current model relies on linear affine transformations and operates on word pairs, future work may explore non-linear mappings and higher-order linguistic structures. The ability to profit from partially observed data makes this approach particularly promising for specialized fields like medical language processing, where it could help infer new relationships in knowledge graphs by "reading" clinical notes.
Summary updated