TL;DR
This paper introduces SETSum, a novel system that uses NLP techniques to automatically summarize and visualize student evaluations of teaching (SETs). By moving beyond static PDF reports, SETSum aims to provide instructors with more efficient, organized, and less biased insights from student feedback, ultimately helping them improve their teaching.
What Problem Does This Paper Address?
Student Evaluations of Teaching (SETs) are a crucial tool in higher education for assessing course quality and instructor performance. However, traditional SET reports, often presented as static PDFs, suffer from several limitations. They typically contain raw, unsorted student comments and basic quantitative statistics, making it difficult for instructors and reviewers to extract meaningful feedback. This lack of organization and summarization can lead to time-consuming analysis, potential cognitive biases (e.g., overemphasizing negative comments), and missed opportunities for instructional improvement. The paper addresses the need for a more dynamic, efficient, and insightful way to present and interpret SET data.
Methods
SETSum employs a multi-stage approach leveraging Natural Language Processing (NLP) and visualization techniques:
- Sentiment Analysis: A sentiment analysis model is trained to classify individual comment sentences as positive or negative. This is achieved by pairing course comments with course ratings and instructor comments with instructor ratings to create training data.
- Aspect Extraction: To identify key themes in student feedback, the system uses MATE (Multi Seed Aspect Extractor), a weakly-supervised model. Unlike LDA-based topic models, MATE uses pre-defined aspects and seed words, aiming for more interpretable and distinct topics.
- Extractive Summarization: For each identified aspect, the system generates a summary by selecting top sentences based on centrality (how representative a sentence is of the aspect), low redundancy, and balanced sentiment. This is an unsupervised, greedy approach.
- Visualization: The system presents quantitative ratings with response rate (circular packing charts) and sentiment distribution (pie charts). For open-ended comments, it uses bubble charts to visualize the prevalence and sentiment of different aspects, with interactive features to drill down into specific comments and summaries.
Datasets
The study utilized Student Evaluations of Teaching (SETs) data collected from the University of North Carolina (UNC) at Chapel Hill over four academic terms (Fall 2017, Spring 2018, Fall 2018, Spring 2019). This dataset comprised approximately 5.6K courses and 298K SETs. For demonstration purposes, the researchers selected two quantitative questions ("Overall, this course was excellent" and "Overall, this instructor was an effective teacher") and two open-ended questions ("Comments on overall assessment of this course" and "Comments on overall assessment of this instructor").
Key Insights and Ideas
- The limitations of static PDF reports are significant: The paper strongly argues that the current standard for SET reporting hinders effective analysis and can even introduce bias.
- NLP can transform SET analysis: By applying sentiment analysis and aspect extraction, the system can automatically identify and categorize feedback, making it much more manageable.
- Visualization is key to understanding: Bubble charts and other visual representations make it easier for instructors to grasp the main themes and sentiments in student comments at a glance.
- Balancing summarization objectives is crucial: The extractive summarization method aims to balance centrality, novelty, and sentiment balance, acknowledging that simply picking the most central sentences isn't always optimal.
- Instructor feedback is vital for system development: The human evaluation component was critical, providing actionable insights that directly informed improvements in SETSUM v1.1.
Key Results
- Automatic Evaluation:
- Sentiment analysis models achieved good performance, with micro F1 scores around 0.83-0.94 for comment-level and 0.83-0.90 for sentence-level predictions.
- Aspect extraction models achieved F1 scores of 48.6 (course) and 48.9 (instructor), with a weakly-supervised RoBERTa-based approach improving these to 62.6 and 64.9 respectively.
- The summarization method showed lower redundancy and sentiment difference compared to a baseline, though at a slight cost to centrality.
- Human Evaluation:
- Ten university professors participated in the evaluation.
- All 10 instructors agreed that SETSUM helps them interpret SET results more efficiently.
- 4 out of 10 instructors felt SETSUM helps them interpret results with less bias.
- 6 out of 10 instructors preferred SETSUM over the standard PDF report, while the remaining 4 would like to have both.
- Instructors found the new SETSUM features to be "useful" to "very useful," with comment analysis features scoring particularly high (average 8.8 out of 10).
Limitations and Future Work
- Sentiment and Aspect Extraction Accuracy: The authors acknowledge that the accuracy of sentiment analysis and aspect extraction models can still be improved, noting that some sentences were inaccurately clustered by aspects.
- Preference for 5-Point Scale: Instructors without a computer science background found concepts like "centrality" difficult to understand. Additionally, some preferred the full 5-point scale for ratings rather than just a positive/negative ratio. These have been addressed in SETSUM v1.1.
- Widespread Use of Aspect Extraction: The current aspect extraction results are noted as not precise enough for widespread use.
- Future Work: The authors plan to:
- Improve sentiment analysis and aspect extraction models.
- Incorporate more functions, such as allowing instructors to compare courses and track their teaching history.
- Develop an administrator dashboard for identifying themes across departments.
- Address co-teaching situations in future versions.
- Explore summarization approaches that align more closely with student sentiment.
Summary updated