Learning PyTorch Through A Neural Dependency Parsing Exercise

David Jurgens2021 · DOI

Abstract

Dependency parsing is increasingly the popular parsing formalism in practice. This assignment provides a practice exercise in implementing the shift-reduce dependency parser of Chen and Manning (2014). This parser is a two-layer feed-forward neural network, which students implement in PyTorch, providing practice in developing deep learning models and exposure to developing parser models.

Introduction

In modern Natural Language Processing (NLP) education, there is a persistent tension between teaching state-of-the-art deep learning and the practical constraints of the classroom. Many contemporary models, such as Transformers, require massive datasets and expensive GPU clusters to train, creating a barrier for students with limited hardware or departments with restricted budgets.

Furthermore, students often struggle to bridge the gap between abstract linguistic concepts—like dependency parsing—and the actual code required to build a neural model from scratch. This paper presents a pedagogical framework designed to address these issues through a "Goldilocks" assignment: a model that is complex enough to be useful and modern, but simple enough to be implemented and trained in a single week on a standard consumer laptop.

The Pedagogical Framework

The assignment centers on the implementation of the Chen and Manning (2014) parser. Unlike modern heavyweights, this model utilizes a relatively simple two-layer feed-forward neural network. By focusing on this architecture, the assignment allows students to master deep learning fundamentals—such as custom activation functions and training loops—without sacrificing the depth of the NLP task.

Technical Implementation

Students are required to implement several key technical components in PyTorch:

  • Shift-Reduce Parsing Logic: Managing the stack and buffer structures used to process a sentence word-by-word.
  • Neural Architecture: Building a network that incorporates word embeddings and a custom "cube" activation function as specified in the original research.
  • The Training Loop: Writing the code that computes loss and applies gradient descent. This "engine" of the model is critical for demystifying how weights are updated.
  • Scaffolding: To keep the focus on neural logic, the author provides scaffolding code to handle data loading and evaluation metrics.

Datasets and Evaluation

The assignment utilizes two primary data resources to ground the exercise in real-world application:

  1. CoNLL Treebank Data: Specifically the 2007 shared task data, which provides gold-standard dependency relations for training.
  2. Pre-trained Embeddings: Students use word vectors from sources like Wikipedia and Twitter. This allows them to observe how different pre-baked representations affect the parser's accuracy and convergence speed, serving as a conceptual bridge to modern Large Language Models (LLMs).

The primary metric for success is the Unlabeled Attachment Score (UAS), giving students a concrete way to measure their model's performance against established NLP standards.

Key Insights and Pedagogical Results

While the focus is pedagogical, the assignment yielded several qualitative and quantitative successes:

  • Demystifying the "Black Box": Students reported that building the training loop from scratch helped them understand that the "magic" of AI is essentially understandable mathematics.
  • Efficiency as an Enabler: Because the feed-forward network trains rapidly, students receive near-instant feedback. This encourages active experimentation with hyperparameters like hidden layer size and dropout.
  • Linguistic Analysis: To prevent students from falling into a "tooling trap"—where they focus solely on code rather than linguistics—the assignment includes a diagnostics requirement. Students must manually analyze incorrect parses to understand the linguistic reasons behind model failure.
  • Accessibility: The model is hardware-agnostic, capable of being trained on laptops several years old, which ensures the curriculum remains inclusive.

Limitations and Future Work

The author identifies a few challenges in the current implementation. Students often struggle with "non-projective" parses (where dependency lines cross), which are a classic hurdle in shift-reduce parsing. Additionally, there is a constant risk that the technical complexity of PyTorch overshadows the linguistic goals of the course.

Future iterations of the assignment may involve having students write the code that converts raw trees into the sequence of shift-reduce operations, providing a deeper look into the data engineering side of NLP and further strengthening the link between linguistic structure and machine learning implementation.

Summary updated

Abstract

Dependency parsing is increasingly the popular parsing formalism in practice. This assignment provides a practice exercise in implementing the shift-reduce dependency parser of Chen and Manning (2014). This parser is a two-layer feed-forward neural network, which students implement in PyTorch, providing practice in developing deep learning models and exposure to developing parser models.
BibTeX
@misc{jurgens2021learning,
  author = {David Jurgens},
  title = {{Learning PyTorch Through A Neural Dependency Parsing Exercise}},
  year = {2021},
  doi = {10.18653/v1/2021.teachingnlp-1.10},
}
Figure