SETSum: Summarization and Visualization of Student Evaluations of Teaching

Yinuo Hu, Shiyue Zhang, Viji Sathy, Abigail T. Panter, Mohit Bansal

Introduction

University instructors face a significant challenge at the end of each semester: interpreting Student Evaluations of Teaching (SETs). While quantitative ratings are easily graphed, qualitative, open-ended comments are typically delivered as a massive, unsorted "wall of text." For large courses, these reports can exceed ten pages, creating hurdles in efficiency and introducing cognitive biases, such as "negativity bias," where instructors fixate on outliers rather than constructive themes. SETSum is a novel system designed to transform these raw comments into interactive, visual dashboards that facilitate less biased and more efficient feedback analysis.

The SETSum Pipeline

The researchers developed a multi-stage NLP pipeline to process raw SET data from the University of North Carolina at Chapel Hill, encompassing approximately 298,000 individual evaluations.

Sentiment Analysis and Aspect Extraction

To categorize feedback, SETSum utilizes a fine-tuned RoBERTa model for sentiment analysis. Because SET data lacks explicit sentiment labels, the system uses students' quantitative Likert-scale ratings as proxy labels. For topic grouping (e.g., "assignments," "exams"), the system employs the Multi Seed Aspect Extractor (MATE), a weakly-supervised model that learns to associate terms based on a few human-provided seed words.

Sentiment-Balanced Summarization

The core technical innovation of SETSum is its extractive summarization algorithm. Unlike standard methods like LexRank, which prioritize the most representative (central) sentences, SETSum optimizes for three criteria:

  1. Centrality: Representativeness of the topic.
  2. Redundancy: Minimizing repetitive information.
  3. Sentiment Balance: Ensuring the summary's sentiment distribution matches the overall sentiment of the source comments, preventing the erasure of minority opinions.

Visualization and Interaction

The front-end provides an interactive experience using bubble charts where the size of each bubble represents comment volume and the color indicates average sentiment.

The SETSum interactive dashboard showing aspect bubbles and a sentiment-balanced summary for the 'Assignment' topic.

Key Results

Model and Summarization Performance

The sentiment analysis model achieved high F1 scores ranging from 0.83 to 0.94. Quantitative evaluation of the summarization algorithm demonstrated that SETSum significantly reduced redundancy and improved sentiment alignment compared to baseline extractive methods.

Comparison of summarization metrics between a baseline and the SETSum approach across Course and Instructor levels.

Human Evaluation

A study involving ten professors across eight departments revealed that 100% of participants found SETSum more efficient for interpreting results than traditional PDF reports. Notably, 40% of instructors specifically highlighted that the system helped them understand their evaluations with reduced cognitive bias.

Conclusion and Future Work

SETSum demonstrates that NLP-driven visualization can mitigate the challenges of qualitative student feedback. While the system effectively reduces bias and increases efficiency, future work will focus on improving aspect extraction precision and adding longitudinal tracking to allow instructors to monitor teaching trends over multiple years. The authors caution that while SETSum is a powerful tool for professional development, it should not be used for high-stakes administrative judgments without acknowledging potential NLP-induced errors.

Summary updated

BibTeX
@inproceedings{hu2022setsum,
  author = {Yinuo Hu and Shiyue Zhang and Viji Sathy and Abigail T. Panter and Mohit Bansal},
  title = {{SETSum: Summarization and Visualization of Student Evaluations of Teaching}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure