Contextual Word Representations: A Contextual Introduction

Noah A. SmitharXiv 2019 · OpenReview

TL;DR

This paper provides a foundational introduction to the evolution of representing words in computers, moving from discrete integer IDs to sophisticated contextual word vectors. It explains how these contextual representations, which capture word meaning based on surrounding text, have revolutionized NLP by enabling models to better understand nuanced language and achieve state-of-the-art results across various tasks.

What Problem Does This Paper Address?

The core problem this paper addresses is how to represent words in a way that captures their meaning and allows computers to process and understand human language effectively. Early NLP systems treated words as discrete entities (like unique IDs or integers), which failed to capture semantic relationships between words (e.g., "king" and "queen" are related). This limitation meant that models couldn't easily share knowledge between similar words, hindering their ability to generalize and understand nuanced language. The paper traces the journey from these discrete representations to more flexible vector-based approaches, culminating in the need for contextual word representations that account for how a word's meaning changes based on its surrounding text.

Methods

The paper outlines a progression of methods for word representation:

  • Discrete Representations: Initially, words were mapped to unique, arbitrary integers. This was computationally efficient but lacked any semantic information.
  • Vector Representations (Word Embeddings): The next step involved representing words as vectors in a multi-dimensional space.
  • "One-hot" vectors: A simple vector where each word gets its own dimension, set to 1, with all others 0. This is still a discrete representation in practice.
  • Feature-based vectors: Dimensions were designed to capture specific linguistic features (e.g., part-of-speech, semantic categories). This required expert knowledge.
  • Distributional Semantics (e.g., Word2Vec): This approach leverages the idea that words appearing in similar contexts have similar meanings. Words are represented by vectors derived from their co-occurrence patterns with other words in a large corpus. Techniques like dimensionality reduction were used to create more compact and efficient vectors.
  • Contextual Word Vectors (e.g., ELMo): This is the most significant advancement discussed. Instead of a single fixed vector for each word type, contextual models generate a unique vector for each word token based on its specific surrounding context. This is achieved by passing word type vectors through neural networks (like LSTMs or Transformers) that process the entire sentence, allowing the representation to adapt to the word's meaning in that particular instance.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation in the introductory sections. However, it frequently refers to the use of large text corpora for deriving word representations, such as:

  • "large text collections on the web"
  • "56M tweets" (in reference to Figure 1)
  • "a science news story" (in reference to Figure 2)

The paper also mentions benchmark tasks and datasets for evaluating contextual word vectors, including:

  • SQuAD (Stanford Question Answering Dataset)
  • Ontonotes (for semantic role labeling and coreference resolution)
  • CoNLL 2003 (for named entity recognition)
  • SWAG (for grounded commonsense reasoning)

Key Insights and Ideas

  • Distributional Hypothesis: The core idea that "words that can be used in similar ways are likely to have related meanings" is fundamental. Observing word co-occurrence patterns in large corpora is a powerful way to infer semantic relationships.
  • Limitations of Discrete Representations: Representing words as simple integers or one-hot vectors fails to capture the rich, continuous spectrum of semantic similarity between words.
  • The Power of Context: A word's meaning is not fixed; it's highly dependent on the surrounding words. Contextual word vectors are crucial for disambiguating word senses and understanding nuanced language.
  • "Man is to computer programmer as woman is to homemaker?": This example highlights how biases present in training data can be encoded into word embeddings, leading to undesirable associations.
  • Analogy Solving: Early successes with word vectors included solving analogies like "man is to woman as king is to queen" by performing vector arithmetic, demonstrating that these representations captured meaningful relationships.
  • Scalability and Pretraining: The development of efficient algorithms (like word2vec) and the ability to pretrain large models on massive corpora have made sophisticated word representations accessible.

Key Results

The paper focuses on explaining the concepts and evolution rather than presenting novel experimental results. However, it references the impact of contextual word vectors on various NLP tasks:

  • ELMo showed significant improvements, including a 9% relative error reduction on the SQuAD benchmark.
  • BERT and subsequent models (like RoBERTa, XLNet) have achieved even greater gains, with BERT showing a 45% error reduction on one task and a 66% relative improvement on SWAG compared to non-contextual methods.
  • These models have consistently demonstrated benefits across a wide range of NLP tasks, including question answering, semantic role labeling, named entity recognition, and coreference resolution.

Limitations and Future Work

  • Bias in Word Vectors: A significant limitation acknowledged is that word vectors, being derived from data, can inherit and perpetuate societal biases present in the text corpora. This is an active area of research.
  • Ambiguity: While contextual vectors help, the paper notes that completely solving the challenge of ambiguous words remains an open question.
  • Interpretability: Reduced-dimensionality vectors, while efficient, can make individual dimensions difficult to interpret.
  • Future Directions: The paper anticipates continued exploration of variations on contextual word vectors, their application to new problems, and modifications to learning methods. It also highlights the potential for these methods in low-resource settings and the need for computationally less expensive approaches. There's also an ongoing effort to characterize what these models are learning and what they are not, and to develop better evaluation benchmarks.

Summary updated

Abstract

This introduction aims to tell the story of how we put words into computers. It is part of the story of the field of natural language processing (NLP), a branch of artificial intelligence. It targets a wide audience with a basic understanding of computer programming, but avoids a detailed mathematical treatment, and it does not present any algorithms. It also does not focus on any particular application of NLP such as translation, question answering, or information extraction. The ideas presented here were developed by many researchers over many decades, so the citations are not exhaustive but rather direct the reader to a handful of papers that are, in the author's view, seminal. After reading this document, you should have a general understanding of word vectors (also known as word embeddings): why they exist, what problems they solve, where they come from, how they have changed over time, and what some of the open questions about them are. Readers already familiar with word vectors are advised to skip to Section 5 for the discussion of the most recent advance, contextual word vectors.
BibTeX
@misc{smith2019contextual,
  author = {Noah A. Smith},
  title = {{Contextual Word Representations: A Contextual Introduction}},
  year = {2019},
  eprint = {1902.06006},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1902.06006},
}
Figure