Introduction
University instructors face a significant challenge at the end of each semester: interpreting Student Evaluations of Teaching (SETs). While quantitative ratings are easily graphed, qualitative, open-ended comments are typically delivered as a massive, unsorted "wall of text." For large courses, these reports can exceed ten pages, creating hurdles in efficiency and introducing cognitive biases, such as "negativity bias," where instructors fixate on outliers rather than constructive themes. SETSum is a novel system designed to transform these raw comments into interactive, visual dashboards that facilitate less biased and more efficient feedback analysis.
The SETSum Pipeline
The researchers developed a multi-stage NLP pipeline to process raw SET data from the University of North Carolina at Chapel Hill, encompassing approximately 298,000 individual evaluations.
Sentiment Analysis and Aspect Extraction
To categorize feedback, SETSum utilizes a fine-tuned RoBERTa model for sentiment analysis. Because SET data lacks explicit sentiment labels, the system uses students' quantitative Likert-scale ratings as proxy labels. For topic grouping (e.g., "assignments," "exams"), the system employs the Multi Seed Aspect Extractor (MATE), a weakly-supervised model that learns to associate terms based on a few human-provided seed words.
Sentiment-Balanced Summarization
The core technical innovation of SETSum is its extractive summarization algorithm. Unlike standard methods like LexRank, which prioritize the most representative (central) sentences, SETSum optimizes for three criteria:
- Centrality: Representativeness of the topic.
- Redundancy: Minimizing repetitive information.
- Sentiment Balance: Ensuring the summary's sentiment distribution matches the overall sentiment of the source comments, preventing the erasure of minority opinions.
Visualization and Interaction
The front-end provides an interactive experience using bubble charts where the size of each bubble represents comment volume and the color indicates average sentiment.
Key Results
Model and Summarization Performance
The sentiment analysis model achieved high F1 scores ranging from 0.83 to 0.94. Quantitative evaluation of the summarization algorithm demonstrated that SETSum significantly reduced redundancy and improved sentiment alignment compared to baseline extractive methods.
Human Evaluation
A study involving ten professors across eight departments revealed that 100% of participants found SETSum more efficient for interpreting results than traditional PDF reports. Notably, 40% of instructors specifically highlighted that the system helped them understand their evaluations with reduced cognitive bias.
Conclusion and Future Work
SETSum demonstrates that NLP-driven visualization can mitigate the challenges of qualitative student feedback. While the system effectively reduces bias and increases efficiency, future work will focus on improving aspect extraction precision and adding longitudinal tracking to allow instructors to monitor teaching trends over multiple years. The authors caution that while SETSum is a powerful tool for professional development, it should not be used for high-stakes administrative judgments without acknowledging potential NLP-induced errors.
Summary updated