TL;DR
This paper introduces Nearest Neighbor Overlap (N2O), a novel, task-agnostic method for comparing sentence embedders by measuring the overlap in their nearest neighbors for a given set of queries. Using N2O, they reveal surprising similarities and differences among various embedders, highlighting the impact of subword information and the distinct nature of contextual models like BERT and GPT.
What Problem Does This Paper Address?
The field of Natural Language Processing (NLP) has seen a surge in methods for creating "sentence embedders" – vector representations of entire sentences. These embedders are crucial for tasks like text clustering, information retrieval, and question answering. However, evaluating and comparing these embedders is challenging. Current methods often rely on specific benchmark tasks (like GLUE) or linguistic probes, which might not reflect how well an embedder performs in real-world, diverse applications. The authors aim to provide a more general, task-agnostic way to understand how similar different sentence embedders are in their "understanding" of semantic similarity, using only unannotated text.
Methods
The core contribution is the Nearest Neighbor Overlap (N2O) metric. For two sentence embedders, $e_A$ and $e_B$, and a corpus $C$, N2O works as follows:
- Sample Queries: A set of sentences (queries) is randomly sampled from the corpus $C$.
- Find Nearest Neighbors: For each query sentence, both embedder $e_A$ and embedder $e_B$ are used to find the $k$ nearest neighbor sentences within the corpus.
- Calculate Overlap: The number of sentences that appear in both nearest neighbor lists is counted.
- Average Overlap: This overlap count is normalized by $k$ and averaged across all query sentences.
The resulting N2O score ranges from 0 (no overlap) to 1 (perfect overlap), indicating how similar the two embedders are in their notion of sentence similarity. The authors emphasize that this method requires only an unannotated corpus and is simple to understand and implement. They also explore the impact of varying $k$ (the number of neighbors) and the query sampling strategy, finding their results to be robust.
Datasets
- Corpus for N2O: The English Gigaword corpus (2010 slice), containing approximately 8 million unique sentences from seven news sources. This corpus is used for sampling queries and finding nearest neighbors.
- Implicit Datasets for Embedder Training: The paper lists various pretrained embedders, which were trained on diverse datasets. Examples include:
- Google News (for word2vec)
- Wikipedia and Gigaword (for GloVe)
- Common Crawl (for FastText)
- Stanford Natural Language Inference (SNLI) dataset (for InferSent)
- Various semantic tasks (for Universal Sentence Encoder)
- BooksCorpus (for BERT and GPT)
- Fine-tuning Dataset: For BERT, the authors fine-tuned a model on the MultiNLI (matched subset) dataset.
Key Insights and Ideas
- N2O as a Universal Comparator: The N2O metric offers a powerful, task-agnostic way to compare embedders, moving beyond task-specific benchmarks. It reveals functional similarity based on how embedders organize semantic space.
- Subword Information Matters: The use of subword information (e.g., in FastText variants) significantly impacts N2O, leading to higher overlap among subword-aware models but also indicating a different way of organizing semantic space compared to models without it.
- BERT and GPT as Outliers: Contextual models like BERT and GPT, especially when using specific token representations (like
[CLS]), show surprisingly low N2O with other embedders. This suggests their pre-training objectives might not directly align with general semantic similarity as captured by nearest neighbors in a large corpus. - Averaging vs. Specific Tokens: Averaging word embeddings (even static ones) often yields higher N2O with other embedders compared to using specific token representations from contextual models.
- Paraphrase Detection Variance: The experiment on query paraphrasing reveals a wide variance in how well different embedders can identify semantically similar sentences, with some models performing significantly better than others.
Key Results
- High Similarity Among Static Embeddings: Variations of the same static word embedding methods (e.g., different GloVe dimensions or corpora) show high N2O values, indicating they learn very similar semantic spaces.
- Subword Impact: FastText models with subword information have high N2O with each other but lower N2O with their non-subword counterparts.
- BERT/GPT Specific Token Outliers: BERT and GPT using
[CLS]or last token representations have low N2O with most other embedders. Averaged BERT/GPT embeddings show higher N2O. - ELMo Consistency: Different ELMo models (varying capacity or training data) show high N2O among themselves, indicating consistent behavior.
- Paraphrase Performance: In the query paraphrasing experiment, ELMo models (especially larger ones) and averaged BERT/GPT embeddings performed well, achieving high MRR (Mean Reciprocal Rank) and a high number of paraphrases found within the top 5 nearest neighbors. For instance,
elmo-orig-5.5bachieved an MRR of 0.910. - Robustness: N2O results are stable across different values of $k$ (number of neighbors) and different query samples, with average Spearman's rank correlation coefficients above 0.99.
Limitations and Future Work
- Computational Cost: While the authors' implementation of exact nearest neighbor search is efficient, N2O computation is linear in corpus size. They suggest approximate nearest neighbor (ANN) methods as a viable option for scaling to even larger corpora.
- Interpretation of "Popular" Neighbors: While N2O quantifies overlap, further qualitative analysis is needed to fully understand why certain sentences are popular neighbors across embedders.
- Future Work: The authors suggest that N2O can be used to situate additional embedders by computing their nearest neighbors and comparing them to the results presented in the paper. They also highlight the potential of ANN for scalable N2O computation.
Summary updated