Abstract
We present a series of six programming assignments designed to teach the design and implementation of modern NLP systems to students ranging from advanced undergraduates to PhD candidates. These assignments emphasize a "full-stack" understanding, moving from hand-coded gradients in linear models to sophisticated neural architectures in PyTorch. The curriculum covers a broad spectrum of techniques, from industry-standard linear models to research-oriented deep learning, while remaining computationally tractable and fully autogradable.
Introduction
Teaching modern Natural Language Processing (NLP) requires balancing high-level library proficiency with a deep understanding of underlying algorithmic mechanics. "Neural-first" approaches often rely on high-level abstractions that obscure optimization details, while traditional linguistics-focused courses may neglect the engineering challenges of state-of-the-art systems. This paper introduces a curriculum that bridges this gap through a sequence of six assignments that build complexity incrementally.
The Six-Assignment Curriculum
The core of the proposed curriculum is a sequence of tasks (A1–A6) that transition students from manual implementation to the use of modern deep learning frameworks.
- A1: Linear Sentiment Analysis: Students implement Stochastic Gradient Descent (SGD) and bag-of-words models from scratch, ensuring they understand the fundamentals of optimization before moving to libraries.
- A2: Neural Sentiment Analysis: The transition to PyTorch begins here, with students building Deep Averaging Networks (DANs) and integrating pre-trained GloVe embeddings.
- A3: Structured Prediction (NER): This assignment focuses on Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs) for Named Entity Recognition, emphasizing linguistic structure and ambiguity.
- A4: Language Modeling: Students implement character-level Recurrent Neural Networks (RNNs), focusing on neural sequence modeling.
- A5: Semantic Parsing: This task involves sequence-to-sequence (seq2seq) models with attention to translate natural language into executable logical forms (e.g., GeoQuery).
- A6: Reading Comprehension: The final assignment tasks students with building a simplified version of models like BiDAF to extract answer spans from the SQuAD dataset.
To keep the focus on algorithmic logic, the code is intentionally "underengineered," avoiding complex Pythonic abstractions that might confuse students.
Curriculum Integration
To support the programming tasks, the curriculum is mapped to standard NLP textbooks, providing students with the necessary theoretical background. This ensures that the practical implementation is grounded in established literature.
Pedagogical Insights and Tractability
A primary challenge in modern NLP education is the computational cost of training large models. The authors achieve tractability by selecting specific tasks—such as character-level language modeling and small-scale semantic parsing—that keep vocabulary sizes small and training times short. This allows students to iterate quickly without requiring massive GPU clusters.
Furthermore, the curriculum addresses the "prior gap": the tendency for students to lack an intuition for expected system behavior. By emphasizing debugging strategies and providing clear performance benchmarks, the assignments help students distinguish between difficult tasks and implementation bugs.
Results and Scalability
The curriculum has been tested on over 300 students across multiple academic levels. Its modular design allows instructors to customize the difficulty; for instance, undergraduate courses might focus on the earlier assignments, while PhD courses engage with the more complex structured prediction and parsing tasks. Additionally, assignments A1 through A5 are fully autogradable via platforms like Gradescope, making them viable for large-scale enrollment.
Limitations
While comprehensive, the curriculum currently lacks explicit implementation of transition-based or chart-based parsing due to engineering complexity. Furthermore, while Transformers are discussed in lectures, they are not yet part of the core implementation requirements to keep the coding burden manageable. Future iterations may also incorporate formal assessments of ethics and bias, which are currently handled through class discussion.
Summary updated