Naive Bayes versus BERT: Jupyter notebook assignments for an introductory NLP course

Jennifer Foster, Joachim WagnerNAACL 2021

TL;DR

This paper introduces two Jupyter notebooks designed to teach introductory Natural Language Processing (NLP) students how to build and fine-tune text classification models. The assignments guide students through implementing a classic Naive Bayes classifier and then fine-tuning a powerful BERT model, providing a practical bridge between foundational and state-of-the-art NLP techniques.

What Problem Does This Paper Address?

Teaching introductory NLP can be challenging, especially when bridging the gap between fundamental concepts and cutting-edge deep learning models. Students often need hands-on experience to truly grasp these techniques. This paper addresses the need for practical, accessible assignments that allow undergraduate students to experiment with both traditional machine learning approaches (like Naive Bayes) and modern transformer-based models (like BERT) for a common NLP task: sentiment analysis. The goal is to equip students with the skills to implement, experiment with, and understand these different approaches.

Methods

The paper describes two core assignments, each built around a Jupyter notebook:

  • Notebook One: Sentiment Polarity with Naive Bayes: This notebook guides students through building a document-level sentiment classifier using a bag-of-words representation and a multinomial Naive Bayes algorithm. It covers essential NLP preprocessing steps, feature extraction (including n-grams), and model training using scikit-learn. Students are encouraged to experiment with different learning algorithms (e.g., logistic regression, decision trees) and feature engineering techniques to improve accuracy.
  • Notebook Two: Sentiment Polarity with BERT: This assignment focuses on fine-tuning BERT for the same sentiment analysis task. It leverages the Hugging Face Transformers library and PyTorch Lightning. The notebook introduces students to BERT's tokenization (subword units, special tokens) and addresses the challenge of BERT's input length limitations by demonstrating strategies like slicing or combining parts of documents. Students are given options to either compare BERT's performance to their Naive Bayes models, fine-tune BERT on a different task, or explore methods to overcome input length constraints.

Datasets

The primary dataset used for both assignments is the Pang and Lee (2004) movie review dataset. This dataset consists of 2,000 movie reviews, pre-tokenized and split into ten cross-validation folds. It's chosen for its relatively small size, making it suitable for quick training in an introductory setting.

Key Insights and Ideas

  • Bridging the Gap: The assignments effectively bridge the gap between traditional NLP methods (Naive Bayes) and modern deep learning (BERT), allowing students to see the evolution and power of different approaches firsthand.
  • Hands-on Learning: Providing students with starter notebooks empowers them to experiment and learn by doing, rather than just passively consuming theoretical knowledge.
  • Flexibility in Learning: Offering students choices in the second assignment (error analysis, new tasks, overcoming limitations) caters to different learning styles and interests, fostering independent research skills.
  • Practical Challenges: The assignments highlight practical challenges in NLP, such as feature engineering for traditional models and handling input length constraints for large transformer models.

Key Results

  • Naive Bayes Performance: The baseline Naive Bayes classifier achieved an accuracy of 83%. Students were able to improve this by approximately 3-5 percentage points through experimentation. Logistic regression, with default hyperparameters, often yielded the best results among the traditional methods.
  • Student Engagement: Most students engaged extensively with the assignments, often following the provided suggestions and exploring various experimental avenues.
  • BERT Adoption: For the second assignment, a majority of students chose to fine-tune BERT on new tasks of their interest (e.g., hate speech detection, sentence acceptability), demonstrating their ability to adapt the model beyond the initial example. Notably, no students opted to focus on overcoming BERT's input length limitations for the movie review dataset.

Limitations and Future Work

  • Code Complexity: Some students found the provided notebook code unnecessarily complicated, particularly its object-oriented nature.
  • Effort vs. Marks: A few students felt the assignments required more effort than the marks awarded.

Future improvements planned include:

  • Streamlining the notebook code to simplify it.
  • Reducing the scope of the assignments.
  • Providing more example notebooks for BERT fine-tuning.

Summary updated

BibTeX
@inproceedings{foster2021naive,
  author = {Jennifer Foster and Joachim Wagner},
  title = {{Naive Bayes versus BERT: Jupyter notebook assignments for an introductory NLP course}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure