TL;DR
This paper introduces SIF (Smooth Inverse Frequency), an embarrassingly simple unsupervised method for creating sentence embeddings. By combining a frequency-based weighted average of word vectors with a basic PCA-based denoising step, the authors produce a baseline that outperforms complex supervised models like RNNs and LSTMs on textual similarity tasks.
What Problem Does This Paper Address?
While word embeddings (like Word2Vec or GloVe) have revolutionized NLP by capturing word-level semantics, representing entire sentences remains a challenge. Researchers often turn to sophisticated, computationally expensive architectures—such as Recurrent Neural Networks (RNNs) or Long Short-Term Memory (LSTM) networks—to model sentence meaning.
The problem is that these complex models often require massive amounts of labeled training data and can struggle with "out-of-domain" tasks (transfer learning). Meanwhile, the simplest approach—averaging the word vectors in a sentence—usually performs poorly because frequent, semantically "thin" words (like the, and, a) drown out the meaningful ones. The authors sought a simple, unsupervised method that could serve as a rigorous baseline for the field, especially when labeled training data is scarce or nonexistent.
Methods: Smooth Inverse Frequency (SIF)
The authors propose a two-step algorithm called SIF (Smooth Inverse Frequency), often denoted in their results as +WR (Weighting and Removal):
- Weighted Averaging: Instead of a simple average, they compute a weighted average of word vectors. The weight for a word $w$ is $\frac{a}{a + p(w)}$, where $a$ is a smoothing parameter and $p(w)$ is the word's frequency in a large corpus. This naturally downweights frequent words and boosts rare, informative ones.
- Common Component Removal: They treat a set of sentence vectors as a matrix and compute its first principal component (using PCA/SVD). They then subtract the projection of each sentence onto this component.
Theoretically, this is justified by a generative model where sentences are produced by a "discourse vector" performing a random walk. The PCA step essentially removes "noise" caused by syntax or common linguistic filler that doesn't contribute to the specific meaning of the sentence.
Datasets
The method was evaluated across a wide array of benchmarks:
- Textual Similarity: 22 datasets, including the SemEval Semantic Textual Similarity (STS) tasks from 2012–2015.
- Supervised Tasks: The SICK (Sentences Involving Compositional Knowledge) dataset for similarity and entailment, and the Stanford Sentiment Treebank (SST).
- Word Vectors: The authors tested their method using GloVe, PSL (Paragram-SL999), and their own SN vectors.
- Corpora for Frequencies: Word frequencies were estimated using diverse sources like Wikipedia, Common Crawl, and political blogs to test robustness.
Key Insights and Ideas
- The Power of Denoising: The most surprising insight is that the first principal component of sentence vectors often represents semantically meaningless "common discourse" (like syntax). Removing it acts as a powerful denoising filter.
- Word2Vec Connection: The authors demonstrate that Word2Vec’s "subsampling" trick is mathematically similar to their SIF weighting, providing a theoretical bridge between word-level heuristics and sentence-level modeling.
- Unsupervised Strength: A completely unsupervised method (GloVe + SIF) can beat highly-tuned supervised models. This suggests that for many tasks, the semantic information is already present in the word vectors; we just need a better way to aggregate them.
Key Results
- Massive Gains: On textual similarity tasks, SIF improved performance over unweighted averaging by 10% to 30%.
- Beating Supervised Models: The unsupervised SIF method (shown as +WR) outperformed sophisticated supervised RNN and LSTM models on most STS tasks.
- Robustness: The method is highly stable. The performance remains consistent across different values of the parameter $a$ and different corpora used to estimate word frequencies.
- State of the Art (at the time): When applied to PSL vectors (a semi-supervised approach), the method achieved the best results on 12 out of 22 similarity tasks, even improving upon previous state-of-the-art embeddings.
Limitations and Future Work
- Sentiment Analysis: SIF does not outperform LSTMs on sentiment tasks (like SST). The authors speculate this is because sentiment often hinges on specific words (like not) that SIF might downweight, and because sentiment can be highly dependent on word order.
- Word Order: As a "bag-of-words" style approach, SIF ignores the sequence of words. It cannot distinguish between "the dog bit the man" and "the man bit the dog."
- Future Directions: The authors suggest exploring ensemble methods that combine the semantic clarity of SIF with the sequence-awareness of RNNs.
Summary updated