kōan: A Corrected CBOW Implementation

Ozan Irsoy, Adrian Benton, Karl StratosarXiv 2020 · OpenReview

Abstract

Mikolov et al. (2013a) observed that continuous bag-of-words (CBOW) word embeddings tend to underperform Skip-gram (SG) embeddings, and this finding has been reported in subsequent works. We find that these observations are driven not by fundamental differences in their training objectives, but more likely on faulty negative sampling CBOW implementations in popular libraries such as the official implementation, word2vec.c, and Gensim. We show that after correcting a bug in the CBOW gradient update, one can learn CBOW word embeddings that are fully competitive with SG on various intrinsic and extrinsic tasks, while being many times faster to train.

Introduction

For nearly a decade, the natural language processing community has operated under the assumption that the Continuous Bag-of-Words (CBOW) architecture for Word2vec is inherently inferior to the Skip-gram (SG) architecture in terms of embedding quality. While CBOW was known to be faster, researchers consistently reported better performance from SG. This paper demonstrates that this performance gap is not a theoretical property of the architectures but is instead the result of a widespread implementation error in popular libraries, including the original word2vec.c, Gensim, and fastText.

The Normalization Bug

The CBOW objective involves predicting a target word from the average of its surrounding context embeddings. Applying the chain rule to this objective dictates that the gradient of the loss with respect to any individual context word's embedding must be scaled by $1/C$, where $C$ is the number of words in the context window.

Investigation of standard implementations revealed that this $1/C$ scaling factor is consistently omitted. While the resulting "buggy" gradient still points in a descent direction, it is sub-optimal and leads to poor convergence. To rectify this, we developed kōan, a corrected C++ implementation of Word2vec that incorporates the proper normalization.

Experimental Results

We evaluated the corrected CBOW implementation against standard SG and buggy CBOW baselines using the Wikipedia corpus and the English C4 dataset.

Training Efficiency and Loss

The corrected implementation significantly improves training dynamics. As shown in Figure 1, the fixed CBOW implementation achieves a consistently lower average loss per token compared to the faulty version throughout the training process.

Average loss per token for buggy vs. fixed CBOW implementations.

In terms of wall-clock time, corrected CBOW is remarkably efficient. Training on the full Wikipedia corpus took less than 73 minutes on a single machine, making it 3.31 times faster than Skip-gram under identical conditions.

Embedding Quality

On intrinsic word similarity and analogy benchmarks, corrected CBOW achieved an average score of 70.8, outperforming both Skip-gram (69.4) and the buggy Gensim CBOW (64.6). Extrinsic evaluations on GLUE tasks and Named Entity Recognition (NER) further confirmed that corrected CBOW is a competitive, high-efficiency alternative to SG.

Analysis: The Exploding Norm Problem

A key consequence of the missing normalization factor is the "exploding norm" problem. Without the $1/C$ scaling, the effective learning rate for source embeddings increases linearly with the context window size. This leads to an artificial inflation of vector norms, which saturates the sigmoid activation functions and effectively halts learning.

Mean embedding norms as a function of context width, highlighting the exploding norm issue in faulty implementations.

As illustrated in Figure 2, the mean embedding norm for the faulty implementation grows significantly as the maximum context width increases, whereas the corrected implementation maintains stable norms.

Conclusion

The perceived inferiority of CBOW was a historical artifact of a gradient calculation error. By restoring the mathematically required normalization, CBOW becomes a superior choice for many applications, offering Skip-gram level quality with significantly reduced training overhead. This finding underscores the importance of rigorous implementation verification, such as finite-difference gradient checks, in machine learning software development.

Summary updated

Abstract

Mikolov et al. (2013a) observed that continuous bag-of-words (CBOW) word embeddings tend to underperform Skip-gram (SG) embeddings, and this finding has been reported in subsequent works. We find that these observations are driven not by fundamental differences in their training objectives, but more likely on faulty negative sampling CBOW implementations in popular libraries such as the official implementation, word2vec.c, and Gensim. We show that after correcting a bug in the CBOW gradient update, one can learn CBOW word embeddings that are fully competitive with SG on various intrinsic and extrinsic tasks, while being many times faster to train.
BibTeX
@misc{irsoy2020koan,
  author = {Ozan Irsoy and Adrian Benton and Karl Stratos},
  title = {{kōan: A Corrected CBOW Implementation}},
  year = {2020},
  eprint = {2012.15332},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.15332},
}
Figure