The Document Vectors Using Cosine Similarity Revisited

Zhang Bingyu, Nikolay ArefyevACL 2022

TL;DR

This paper revisits a previously reported state-of-the-art document embedding model, uncovering an error in its evaluation that significantly inflated its accuracy. After correcting this, the authors demonstrate that while large Transformer models like RoBERTa excel with ample data, the simpler DV-ngrams-cosine model can outperform them on very small datasets. They also introduce a novel sub-sampling technique to improve the training of DV-ngrams-cosine.

What Problem Does This Paper Address?

The paper tackles the surprising performance of a relatively simple document embedding model, "Document Vectors using Cosine Similarity" (DV-ngrams-cosine), which had reported an exceptionally high accuracy (97.42%) on the IMDB movie review dataset. This accuracy surpassed even large, pre-trained Transformer models, which is unusual given their general superiority. The authors aimed to understand this discrepancy and investigate the model's true capabilities, especially in comparison to modern Transformer architectures. They also sought to improve the training efficiency and effectiveness of the DV-ngrams-cosine model.

Methods

The core of the paper involves a re-evaluation of the DV-ngrams-cosine model and its ensemble with NB-weighted Bag-of-N-grams (BON) vectors. The authors identified a critical bug in the original evaluation procedure where the document vectors for training and testing were incorrectly concatenated from different documents belonging to the same class, leading to an inflated accuracy.

Key methods include:

  • Error Correction: Identifying and fixing the bug in the evaluation pipeline.
  • Comparative Analysis: Evaluating DV-ngrams-cosine, NB-weighted BON, their ensemble, and the RoBERTa Transformer model across varying training data sizes.
  • NB Sub-Sampling: A novel technique for training DV-ngrams-cosine. This method uses Naive Bayesian weights to prioritize sentiment-related n-grams during the embedding learning process, aiming for faster training and better quality by focusing on more informative features.
  • Ensembling: Exploring combinations of DV-ngrams-cosine with RoBERTa.

Datasets

  • IMDB Movie Reviews Dataset: This dataset was used for both training and evaluating all models. The paper mentions using different splits, including the original one and a train/validation/test split from Suchin et al. (2020).

Key Insights and Ideas

  • The "SOTA" Was an Artifact: The exceptionally high accuracy of the DV-ngrams-cosine ensemble was due to an evaluation bug, not inherent superiority. This highlights the importance of rigorous evaluation protocols.
  • Small Data Advantage: Simpler models like DV-ngrams-cosine can outperform powerful Transformer models (like RoBERTa) when the amount of labeled training data is extremely scarce (e.g., 10-20 documents). This suggests that for low-resource scenarios, simpler, domain-specific models might be more practical.
  • NB Sub-Sampling for Efficiency: The proposed Naive Bayesian sub-sampling technique significantly speeds up the training of DV-ngrams-cosine without introducing overfitting, leading to better performance. This is a valuable optimization for this class of models.
  • Ensembling Nuances: While ensembling DV-ngrams-cosine with RoBERTa yielded marginal improvements, the paper suggests that the effectiveness of ensembling depends heavily on how the different representations are combined, with schemes that implicitly use test labels showing better results (though this is problematic for true generalization).

Key Results

  • Corrected Accuracy: The originally reported 97.42% accuracy for the DV-ngrams-cosine + NB-weighted BON ensemble was corrected to 93.68%. This is only a 0.55% improvement over the pure DV-ngrams-cosine model.
  • RoBERTa vs. DV-ngrams-cosine:
    • With large training sets (e.g., 20,000 examples), RoBERTa generally achieves higher test accuracy.
  • With very small training sets (10 or 20 documents), DV-ngrams-cosine shows higher mean test accuracy and lower standard deviation than RoBERTa.
  • NB Sub-Sampling Impact: The NB sub-sampling scheme leads to faster training progress and improved performance compared to the vanilla training process for DV-ngrams-cosine.
  • Ensemble with RoBERTa: Combining DV-ngrams-cosine (with or without NB sub-sampling) with RoBERTa resulted in a small improvement of 0.13-0.15%.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the NB-weighted BON vectors are weak on their own.
  • The original ensemble's bug highlights a general challenge in ensuring correct evaluation methodologies.
  • The paper notes that while RoBERTa was pre-trained on a general corpus, domain adaptation techniques could potentially improve its performance on smaller datasets, but they chose not to employ them for a "standard approach" comparison.
  • Future Work:
    • The authors suggest that the NB sub-sampling scheme can be used as an "add-on" to the original DV-ngrams-cosine model, implying further exploration of its integration and benefits.
  • The paper's findings on small-data performance suggest further research into why simpler models can excel in such scenarios and how to best leverage them.

Summary updated

BibTeX
@inproceedings{bingyu2022document,
  author = {Zhang Bingyu and Nikolay Arefyev},
  title = {{The Document Vectors Using Cosine Similarity Revisited}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure