Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing

Antoine Bordes, Xavier Glorot, Jason Weston, Yoshua Bengio

Abstract

Open-text semantic parsers are designed to interpret any statement in natural language by inferring a corresponding meaning representation (MR - a formal representation of its sense). Unfortunately, traditional semantic parsing often struggles with the scale of open-domain vocabulary and the ambiguity of natural language. This paper introduces a large-scale semantic parser that maps natural language to disambiguated logical representations by jointly learning from both structured knowledge bases (like WordNet) and raw text (Wikipedia).

Introduction

Semantic parsing—the process of translating natural language into a formal Meaning Representation (MR)—traditionally faces a major data bottleneck. Research has historically been split into two camps: deep, in-domain parsing restricted to tiny vocabularies due to the need for expert manual annotation, and shallow, open-domain parsing capable of handling any text but producing noisy, weakly supervised representations.

This work aims to bridge this gap by creating a system that handles the vast vocabulary of "open-text" while grounding it in a structured dictionary of entities. The core challenge lies in word-sense disambiguation: teaching a machine to map an ambiguous word (like "score") to its specific sense (e.g., musical notation vs. sports result) when labeled examples are scarce.

Methodology

The authors propose a two-stage pipeline consisting of structure inference and entity detection. First, Semantic Role Labeling (SRL) is used to identify the "skeleton" of a sentence, typically a relation(subject, object) triplet of lemmatized words. Second, a Semantic Matching Energy (SME) function maps these lemmas to specific WordNet synsets.

The Semantic Matching Energy Model

The model embeds every lemma, synset, and relation into a $d$-dimensional vector space. To determine the plausibility of a triplet (subject, relation, object), the model utilizes bilinear layers to combine the embeddings of the subject and relation ($g_{left}$) and the object and relation ($g_{right}$). The "energy" (score) is then computed via a dot product ($h(.)$) of these two combinations.

The Semantic Matching Energy (SME) model architecture for scoring semantic triplets.

Lower energy indicates a more plausible semantic match. During inference, the system searches for the combination of WordNet synsets that minimizes this energy for a given sentence. By training on "Bridge" triplets—where one part is a lemma and the other is a synset—the model learns to align raw text with formal logic.

Data and Training

The system is trained simultaneously on multiple data sources to create a shared embedding space:

  • WordNet 3.0 & ConceptNet 2.1: Providing structured dictionary definitions and common-sense knowledge.
  • Wikipedia: 50,000 articles used as raw text, with triplets extracted via SRL.
  • Extended WordNet (XWN) & Unambiguous Wikipedia (Wku): Used for evaluation and synthetic "silver" supervision.

This joint learning approach allows the model to treat every verb in open text as a potential relation, generalizing common-sense knowledge to thousands of different actions.

Experimental Results

The model was evaluated on Word Sense Disambiguation (WSD) and knowledge acquisition. On the XWN test set, the "All+MFS" model—which combines all training sources with a "Most Frequent Sense" heuristic—achieved an F1 score of 72.35%, significantly outperforming models trained on WordNet alone (~29%).

Performance comparison of various models on WordNet and Word Sense Disambiguation tasks.

Qualitatively, the embeddings successfully grouped related concepts. For example, the closest neighbors to the lemma "mark" included synsets for "score," "marking," and "evaluation." While the model showed a slight "degrading effect" on specialized WordNet relations as more open-text verbs were added, it remained the only model capable of scaling to the thousands of relations found in open text.

Conclusion

By embedding words, entities, and relations into a shared vector space using an energy-based model, this system performs word-sense disambiguation and knowledge acquisition within a unified framework. Future work may address current limitations, such as the reliance on shallow subject-verb-object structures and the dependency on initial SRL tools, to move toward more complex, end-to-end unsupervised semantic parsing.

Summary updated

Abstract

Open-text semantic parsers are designed to interpret any statement in natural language by inferring a corresponding meaning representation (MR - a formal representation of its sense). Unfortunately...
BibTeX
@inproceedings{bordes2012joint,
  author = {Antoine Bordes and Xavier Glorot and Jason Weston and Yoshua Bengio},
  title = {{Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2012},
  url = {https://openreview.net/forum?id=STzfe4kIBlq},
}
Figure