Introduction
The Smooth Inverse Frequency (SIF) method, introduced by Arora et al. (2017), has become a staple baseline for generating sentence embeddings. By taking a weighted average of word vectors and removing the first principal component, SIF achieves impressive results on Semantic Textual Similarity (STS) tasks. However, this student abstract presented at AAAI-20 challenges the theoretical "scaffolding" of SIF, arguing that its mathematical justification is built on inconsistent logic and flawed derivations.
Theoretical Inconsistencies
The authors identify two primary mathematical issues within the original SIF generative model that suggest the theory does not align with its empirical implementation.
The Unigram Paradox
The SIF model assumes a generative process where word production depends on a context vector. However, the authors demonstrate that for typical vocabulary sizes, the model's hyperparameters imply a probability $\alpha \approx 0.99999999$. This effectively reduces the complex generative model to a simple unigram model that ignores the sentence's context vector entirely. This contradicts the very purpose of the embedding, which is to capture sentence-level context.
The Taylor Expansion Flaw
The original SIF derivation relies on a Taylor expansion to linearize the log-likelihood. The authors point out a fundamental contradiction: the expansion is performed around a point where the context vector $\tilde{c} \approx 0$. Yet, the model is subsequently optimized on a unit sphere where $\|\tilde{c}\| = 1$. At this distance from the origin, the Taylor approximation is no longer valid, rendering the resulting mathematical justification unsound.
The Dominance of Principal Component Removal (PCR)
A key component of SIF is the removal of the first principal component, which is thought to represent "common discourse." The authors investigate whether SIF's success is due to its specific weighting scheme or this post-processing step.
By comparing SIF against a "Simple Average + PCR" baseline, they found that much of the performance gain attributed to SIF actually comes from the PCR step. While SIF+PCR remains a superior baseline, the gap between it and a simple average narrows significantly once PCR is applied to both, as seen in the SGNS embedding results.
Empirical Evaluation
To further test the validity of the SIF generative model, the authors attempted to derive a "corrected" version, referred to as Model $M$. If the generative theory were sound, this model should theoretically perform well. However, as shown in the table below, Model $M$ failed to outperform standard Pointwise Mutual Information (PMI) scores on word similarity and analogy tasks.
These results suggest that the generative model itself may be an inadequate starting point for explaining the empirical success of SIF.
Conclusion
While SIF remains a robust and "tough-to-beat" empirical baseline, this critique highlights a significant gap between its performance and its theoretical justification. The success of SIF appears to be driven more by sound intuition—such as downweighting frequent words and removing common components—than by the specific generative theory proposed in its original derivation. This work serves as a cautionary reminder that empirical success does not always validate the underlying theoretical framework.
Summary updated