A Simple but Tough-to-Beat Baseline for Sentence Embeddings

Sanjeev Arora, Yingyu Liang, Tengyu MaICLR 2017 · OpenReview

TL;DR

This paper introduces SIF (Smooth Inverse Frequency), an embarrassingly simple unsupervised method for creating sentence embeddings. By combining a frequency-based weighted average of word vectors with a basic PCA-based denoising step, the authors produce a baseline that outperforms complex supervised models like RNNs and LSTMs on textual similarity tasks.

What Problem Does This Paper Address?

While word embeddings (like Word2Vec or GloVe) have revolutionized NLP by capturing word-level semantics, representing entire sentences remains a challenge. Researchers often turn to sophisticated, computationally expensive architectures—such as Recurrent Neural Networks (RNNs) or Long Short-Term Memory (LSTM) networks—to model sentence meaning.

The problem is that these complex models often require massive amounts of labeled training data and can struggle with "out-of-domain" tasks (transfer learning). Meanwhile, the simplest approach—averaging the word vectors in a sentence—usually performs poorly because frequent, semantically "thin" words (like the, and, a) drown out the meaningful ones. The authors sought a simple, unsupervised method that could serve as a rigorous baseline for the field, especially when labeled training data is scarce or nonexistent.

Methods: Smooth Inverse Frequency (SIF)

The authors propose a two-step algorithm called SIF (Smooth Inverse Frequency), often denoted in their results as +WR (Weighting and Removal):

  1. Weighted Averaging: Instead of a simple average, they compute a weighted average of word vectors. The weight for a word $w$ is $\frac{a}{a + p(w)}$, where $a$ is a smoothing parameter and $p(w)$ is the word's frequency in a large corpus. This naturally downweights frequent words and boosts rare, informative ones.
Weighting function comparison showing how SIF downweights frequent words.
  1. Common Component Removal: They treat a set of sentence vectors as a matrix and compute its first principal component (using PCA/SVD). They then subtract the projection of each sentence onto this component.

Theoretically, this is justified by a generative model where sentences are produced by a "discourse vector" performing a random walk. The PCA step essentially removes "noise" caused by syntax or common linguistic filler that doesn't contribute to the specific meaning of the sentence.

Datasets

The method was evaluated across a wide array of benchmarks:

  • Textual Similarity: 22 datasets, including the SemEval Semantic Textual Similarity (STS) tasks from 2012–2015.
  • Supervised Tasks: The SICK (Sentences Involving Compositional Knowledge) dataset for similarity and entailment, and the Stanford Sentiment Treebank (SST).
  • Word Vectors: The authors tested their method using GloVe, PSL (Paragram-SL999), and their own SN vectors.
  • Corpora for Frequencies: Word frequencies were estimated using diverse sources like Wikipedia, Common Crawl, and political blogs to test robustness.

Key Insights and Ideas

  • The Power of Denoising: The most surprising insight is that the first principal component of sentence vectors often represents semantically meaningless "common discourse" (like syntax). Removing it acts as a powerful denoising filter.
  • Word2Vec Connection: The authors demonstrate that Word2Vec’s "subsampling" trick is mathematically similar to their SIF weighting, providing a theoretical bridge between word-level heuristics and sentence-level modeling.
  • Unsupervised Strength: A completely unsupervised method (GloVe + SIF) can beat highly-tuned supervised models. This suggests that for many tasks, the semantic information is already present in the word vectors; we just need a better way to aggregate them.

Key Results

  • Massive Gains: On textual similarity tasks, SIF improved performance over unweighted averaging by 10% to 30%.
  • Beating Supervised Models: The unsupervised SIF method (shown as +WR) outperformed sophisticated supervised RNN and LSTM models on most STS tasks.
Performance comparison on textual similarity tasks.
  • Robustness: The method is highly stable. The performance remains consistent across different values of the parameter $a$ and different corpora used to estimate word frequencies.
  • State of the Art (at the time): When applied to PSL vectors (a semi-supervised approach), the method achieved the best results on 12 out of 22 similarity tasks, even improving upon previous state-of-the-art embeddings.

Limitations and Future Work

  • Sentiment Analysis: SIF does not outperform LSTMs on sentiment tasks (like SST). The authors speculate this is because sentiment often hinges on specific words (like not) that SIF might downweight, and because sentiment can be highly dependent on word order.
  • Word Order: As a "bag-of-words" style approach, SIF ignores the sequence of words. It cannot distinguish between "the dog bit the man" and "the man bit the dog."
  • Future Directions: The authors suggest exploring ensemble methods that combine the semantic clarity of SIF with the sequence-awareness of RNNs.

Summary updated

Abstract

The success of neural network methods for computing word embeddings has motivated methods for generating semantic embeddings of longer pieces of text, such as sentences and paragraphs. Surprisingly, Wieting et al (ICLR'16) showed that such complicated methods are outperformed, especially in out-of-domain (transfer learning) settings, by simpler methods involving mild retraining of word embeddings and basic linear regression. The method of Wieting et al. requires retraining with a substantial labeled dataset such as Paraphrase Database (Ganitkevitch et al., 2013). The current paper goes further, showing that the following completely unsupervised sentence embedding is a formidable baseline: Use word embeddings computed using one of the popular methods on unlabeled corpus like Wikipedia, represent the sentence by a weighted average of the word vectors, and then modify them a bit using PCA/SVD. This weighting improves performance by about 10% to 30% in textual similarity tasks, and beats sophisticated supervised methods including RNN's and LSTM's. It even improves Wieting et al.'s embeddings. This simple method should be used as the baseline to beat in future, especially when labeled training data is scarce or nonexistent. The paper also gives a theoretical explanation of the success of the above unsupervised method using a latent variable generative model for sentences, which is a simple extension of the model in Arora et al. (TACL'16) with new "smoothing" terms that allow for words occurring out of context, as well as high probabilities for words like and, not in all contexts.
BibTeX
@inproceedings{arora2017simple,
  author = {Sanjeev Arora and Yingyu Liang and Tengyu Ma},
  title = {{A Simple but Tough-to-Beat Baseline for Sentence Embeddings}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=SyK00v5xx},
}
Figure