LSTMEmbed: Learning Word and Sense Representations from a Large Semantically Annotated Corpus with Long Short-Term Memories

Ignacio Iacobacci, Roberto NavigliACL 2019 · DOI

Abstract

Traditional word embedding models often conflate multiple meanings of a word into a single vector. LSTMEmbed addresses this by learning distinct representations for words and their specific senses simultaneously. By leveraging a bidirectional Long Short-Term Memory (BiLSTM) network trained on a massive, sense-annotated corpus and utilizing pretrained embeddings as an optimization target, LSTMEmbed captures both sequential context and deep semantic knowledge. The model achieves state-of-the-art performance on word-to-sense similarity tasks and demonstrates superior accuracy in synonym recognition.

Introduction

Most popular word embedding models, such as word2vec or GloVe, represent each word with a single vector regardless of its polysemy. For instance, the word "bank" is typically represented by a single vector that averages its financial and geographical meanings. While sense embeddings have been proposed to solve this "conflation" problem, many existing approaches treat sentences as "bags of words," ignoring the critical information provided by word order. Furthermore, many models fail to represent words and senses in a unified geometric space, complicating direct comparisons between a word and its specific meanings.

Methods

LSTMEmbed introduces a bidirectional LSTM architecture designed to process sense-annotated text. The model takes a corpus where words are tagged with specific sense identifiers (e.g., BabelNet IDs). Unlike window-based models, the BiLSTM processes the preceding and following context of a target word in sequence, capturing how syntactic structure and word order influence meaning.

Architecture of LSTMEmbed

A key innovation of LSTMEmbed is its training objective. Instead of predicting a word's identity, the model is trained to produce a vector that minimizes the cosine distance to a pretrained embedding of the target token. This "distillation" approach allows the model to inherit semantic knowledge from high-quality static embeddings while remaining sequence-aware. Both words and senses are learned within a single look-up table, ensuring they share the same vector space for direct mathematical comparison.

Datasets and Evaluation

The model was trained on BabelWiki, a version of Wikipedia automatically annotated with senses using the Babelfy tool, comprising approximately 3 billion tokens. BabelNet (v4.0) provided the underlying sense inventory. Evaluation spanned several benchmarks, including SemEval-2014 (Task 3) for word-to-sense similarity, TOEFL & ESL-50 for synonym recognition, and the 8-8-8 dataset for outlier detection.

Key Results

LSTMEmbed achieved state-of-the-art (SOTA) performance on the SemEval-2014 word-to-sense similarity task, reaching a Spearman correlation of 0.400. This result outperformed previous leading models such as MeerkatMafia and DeConf.

Performance on SemEval-2014 word-to-sense similarity

In synonym recognition, the model achieved 92.50% accuracy on the TOEFL dataset, significantly surpassing standard word2vec (87.00%) and GloVe (88.75%). The model also proved robust in outlier detection, ranking second only to SensEmbed while maintaining superior performance across general word-level tasks. Analysis indicates that the quality of the final embeddings is directly influenced by the richness of the pretrained embeddings used as the training target.

Discussion and Future Work

By integrating sequence modeling with sense-level annotations, LSTMEmbed effectively bridges the gap between word-level and sense-level semantics. The unified vector space allows for nuanced queries, such as comparing a specific sense of a word to general concepts. However, the model faces limitations, such as dimensionality constraints and a dependency on the accuracy of automated sense-tagging tools. Future research could explore replacing the BiLSTM with Transformer-based architectures to further enhance contextual representation.

Summary updated

BibTeX
@inproceedings{iacobacci2019lstmembed,
  author = {Ignacio Iacobacci and Roberto Navigli},
  title = {{LSTMEmbed: Learning Word and Sense Representations from a Large Semantically Annotated Corpus with Long Short-Term Memories}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1165},
}
Figure