Learning How To Learn NLP: Developing Introductory Concepts Through Scaffolded Discovery

Alexandra Schofield, Richard Wicentowski, Julie MederoNAACL 2021

TL;DR

This paper introduces a "scaffolded discovery learning" approach for teaching introductory Natural Language Processing (NLP) to undergraduate students. Instead of focusing on implementing specific algorithms upfront, this method emphasizes developing core analytical skills like data exploration, quantitative analysis, and understanding model limitations, which the authors argue better prepares students for future research and innovation in NLP.

What Problem Does This Paper Address?

Traditional NLP courses often jump straight into teaching established algorithms, which can lead students to focus on achieving performance metrics without a deep understanding of the underlying linguistic principles or why certain choices are made. This approach can be problematic because:

  • It misses the "why": Students might implement algorithms without grasping the linguistic rationale behind them, hindering their ability to adapt to new domains or problems.
  • The NLP landscape shifts rapidly: What's considered "state-of-the-art" changes quickly, making deep dives into specific, rapidly evolving models potentially less valuable in the long run.
  • It can be less accessible: Undergraduate NLP courses often have students with diverse backgrounds in math, computer science, and linguistics, making a purely algorithmic approach challenging.

The authors aim to address this by proposing a pedagogical framework that prioritizes foundational analytical skills over immediate algorithmic implementation.

Methods

The core of the paper is the description of a pedagogical approach centered around scaffolded discovery learning. This method involves:

  • Broad Objectives: Students are given a general goal or problem to solve.
  • Guided Exploration: Instead of direct instruction, students are provided with scaffolding (e.g., starter code, clear instructions, checkpoints) to help them explore and "discover" the necessary concepts and skills.
  • Emphasis on Analysis: The assignments are designed to encourage students to analyze quantitative results, explore textual data to understand model behavior, and think critically about model limitations.
  • De-emphasis on Specific Models: The labs focus on generalizable skills rather than requiring mastery of specific, cutting-edge NLP models.

The authors detail four specific lab assignments designed for the first four weeks of an NLP course, each building on the previous one and introducing core NLP concepts through this discovery-based framework. These labs are implemented in Python with starter code provided via GitHub and often utilize autograders for immediate feedback.

Datasets

The paper mentions the use of:

  • Project Gutenberg texts: Used for tasks involving word frequencies, Zipf's Law, and n-gram models.
  • Brown corpus: Used for sentence segmentation.
  • Hyperpartisan news dataset (Kiesel et al., 2019): Used to explore unique features in different types of news.
  • GloVe embeddings (Pennington et al., 2014): Used for experiments with word vector relationships and analogies.

The authors do not introduce new benchmarks but rather leverage existing, publicly available datasets to illustrate their pedagogical approach.

Key Insights and Ideas

  • Discovery learning can be effective when scaffolded: Pure discovery learning can be ineffective, but providing structured guidance allows students to develop deeper understanding and ownership of their learning.
  • Analytical skills are more transferable than specific algorithms: Skills like data navigation, experiment design, and critical evaluation of results are more enduring and applicable across various NLP tasks and future research directions than memorizing specific model architectures.
  • Focusing on "why" fosters better problem-solving: Understanding the linguistic rationale behind NLP techniques allows students to adapt and innovate, rather than just applying pre-learned solutions.
  • Liberal arts approach to NLP: The method encourages students to critique the nature of NLP evaluation and consider the societal implications of language technologies.
  • Early exposure to data nuances is crucial: Labs like tokenization and segmentation help students appreciate the inherent variability and "messiness" of real-world text data, which is often obscured in benchmark results.

Key Results

The paper focuses on the pedagogical outcomes rather than quantitative performance metrics of NLP models. The authors report that:

  • Students develop the ability to perform standard string manipulations, analyze quantitative results, and explore textual data.
  • They gain experience in identifying model limitations and proposing improvements.
  • The approach helps students develop intuitions about language data and the challenges of NLP.
  • Students are better prepared to engage with more complex topics and student-led projects later in the semester.
  • The labs are designed to be self-contained, re-introducing necessary concepts and formulas to ensure accessibility.

Limitations and Future Work

  • Risk of students getting stuck: The authors acknowledge that discovery learning can lead to students becoming confused or stuck, and they suggest that more guidance on file I/O and analysis scope might be beneficial.
  • Less technical depth on modern models: While the approach prepares students for understanding modern models, it doesn't delve deeply into their implementation details early on. The authors suggest that sample write-ups and refined prompts can help students provide more thoughtful analyses.
  • Future work: The authors have made their assignment instructions and lab files publicly available on GitHub and are willing to share additional autograding files, encouraging others to adopt and adapt their approach. They also suggest further development of prompts to elicit more insightful analyses.

Summary updated

BibTeX
@inproceedings{schofield2021learning,
  author = {Alexandra Schofield and Richard Wicentowski and Julie Medero},
  title = {{Learning How To Learn NLP: Developing Introductory Concepts Through Scaffolded Discovery}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure