Learning about Word Vector Representations and Deep Learning through Implementing Word2vec

David Jurgens2021 · DOI

Abstract

Word vector representations are an essential part of an NLP curriculum. Here, we describe a homework that has students implement a popular method for learning word vectors, word2vec. Students implement the core parts of the method, including text preprocessing, negative sampling, and gradient descent.

Introduction: The "Black Box" Problem in NLP Education

In modern Natural Language Processing (NLP) education, students frequently rely on high-level deep learning libraries such as PyTorch or TensorFlow. While these tools are indispensable for production-level development, they often abstract away the underlying mechanics of model training. This "black box" approach can lead to a superficial understanding of how algorithms actually function.

Existing curricula often fall into two extremes: "Applied" courses that focus on using pre-trained models, or "Theoretical" courses that focus heavily on mathematical proofs. This paper introduces a middle-ground pedagogical approach: an assignment that requires students to ground formal mathematical descriptions of the word2vec algorithm into functional software operations using only numpy. By bridging this gap, students develop a "mechanical" intuition for gradient descent, negative sampling, and the nuances of text preprocessing.

Methods: Building Word2vec from Scratch

The core of the assignment is the implementation of the Skip-gram model from the ground up. To ensure students engage with the underlying logic, the use of high-level deep learning frameworks is prohibited.

1. Manual Implementation with Numpy

Students must use numpy for all numerical operations. This constraint serves a dual purpose: it forces students to understand the linear algebra involved and highlights the critical importance of vectorization. Students quickly learn that standard Python for loops are insufficient for the computational demands of word vector training, necessitating efficient matrix operations.

2. Text Preprocessing

Before training begins, students implement essential preprocessing steps:

  • Low-frequency token removal: Cleaning the vocabulary to focus on statistically significant words.
  • Subsampling: Implementing frequency-based subsampling to transform raw text into self-supervised training examples, reducing the influence of overly frequent words (like "the" or "and").

3. Core Training Procedure

The heart of the assignment involves three technical pillars:

  • Negative Sampling: Students generate "noise" examples to make the training process computationally feasible, avoiding the need to update every word in the vocabulary for every training instance.
  • Gradient Descent: Students manually calculate and apply updates to two distinct word vector matrices (input and output vectors).
  • Loss Calculation: Students compute the negative log-likelihood (NLL) to monitor the model's convergence over time.

4. Evaluation

Once the vectors are trained, students utilize the Gensim library to load their custom weights. They perform qualitative tests, such as word analogies (e.g., "man is to king as woman is to X") and nearest-neighbor searches, alongside quantitative similarity scoring.

Key Pedagogical Insights

The "Equation-to-Code" Hurdle

The most significant learning moment identified is the translation of loss functions and gradient equations into executable code. While students often find this "grounding" exercise daunting, they report it as the most rewarding aspect of the curriculum. It demystifies the transition from a LaTeX equation in a research paper to a functioning system.

Debugging via Semantics

A unique aspect of this assignment is that students learn to debug their mathematics through semantic output. If, after several minutes of training, the nearest neighbors for "January" are not other months or seasonal terms, students can infer a bug in their gradient descent logic. This provides a tangible feedback loop that pure mathematical proofs lack.

Type vs. Token Confusion

The assignment highlights common conceptual stumbling blocks, such as the distinction between word types (unique vocabulary entries) and tokens (individual instances in a sequence). Errors frequently arise when students accidentally subsample types instead of tokens during the preprocessing phase, providing a practical lesson in data handling.

Results and Benchmarking

The assignment utilizes Kaggle InClass for benchmarking. Students upload their similarity scores—calculated using a subset of the SimLex-999 gold-standard dataset—to a leaderboard. This competitive element serves as a diagnostic tool; low scores often help students identify subtle implementation bugs that do not cause code crashes but result in poor vector quality.

Overall, students reported a significant boost in confidence regarding their ability to read and understand original NLP research papers. The assignment successfully prepares them for more advanced concepts, such as Transformer-based models (e.g., BERT), by providing a foundational intuition for what modern libraries do "under the hood."

Limitations and Future Work

While effective, the author notes that the assignment's mathematical depth might be considered shallow for a dedicated Machine Learning course, which might require more rigorous derivations.

Future iterations of the assignment could include:

  • Bias and Ethics: Using Wikipedia biographies as a corpus to identify and mitigate gender or racial biases encoded in learned vectors.
  • Algorithmic Extensions: Implementing features like dropout, learning rate decay, or the incorporation of external knowledge bases to further refine vector quality.

Summary updated

Abstract

Word vector representations are an essential part of an NLP curriculum. Here, we describe a homework that has students implement a popular method for learning word vectors, word2vec. Students implement the core parts of the method, including text preprocessing, negative sampling, and gradient descent.
BibTeX
@misc{jurgens2021learning,
  author = {David Jurgens},
  title = {{Learning about Word Vector Representations and Deep Learning through Implementing Word2vec}},
  year = {2021},
  doi = {10.18653/v1/2021.teachingnlp-1.19},
}
Figure