Practical Sentiment Analysis for Education: The Power of Student Crowdsourcing

Robert Kasumba, Marion NeummanAAAI 2024 · DOI

TL;DR

This paper explores using student-generated labels from crowdsourcing to train sentiment analysis models for educational feedback. They found that these crowdsourced labels perform comparably to human expert labels, offering a more scalable and cost-effective solution for understanding student sentiment in large courses.

What Problem Does This Paper Address?

As university enrollment, particularly in computer science, continues to rise, so does the gap between the volume of student feedback and the capacity of instructors to analyze it. While free-form text reflections (Unit-of-Study Reflections or USRs) offer rich insights into student experiences, manually reading and analyzing them is time-consuming. Existing automated sentiment analysis methods often struggle with domain-specific language, pre-trained models can be biased, and dictionary-based approaches lack contextual understanding. The core challenge is to find an efficient and reliable way to automatically analyze this valuable student feedback to improve teaching and identify struggling students.

Methods

The researchers employed supervised machine learning (ML) approaches, specifically Random Forest (RF) and Support Vector Machines (SVM), for sentiment analysis. The novelty lies in their investigation of using student-generated labels for training these models. They compared three types of labels:

  • Self-Reported (SR) labels: Students' own 5-star ratings of their reflections.
  • Crowdsourced (CS) labels: Labels collected from other students in a separate data science course, who were incentivized to provide ratings.
  • Combined (CS+SR) labels: A combination of crowdsourced and self-reported labels.

They also developed a label aggregation process using a weighted majority vote (WMV) with an expectation-maximization (EM) algorithm to account for varying labeler accuracy. For feature extraction, they primarily used TF-IDF with unigrams and bigrams, and also experimented with BERT embeddings, finding TF-IDF to be more effective. A hybrid approach incorporating VADER (Valence Aware Dictionary and sentiment Reasoner) scores with TF-IDF features also showed improvement.

Datasets

The study utilized several datasets of student homework reflections (USRs) collected from eight large computer science courses at Washington University in St. Louis between Spring 2016 and 2020.

  • FL18 and FL19 datasets: These were used for the crowdsourcing experiment. FL18 also had human expert-annotated (HE) labels available for ground truth.
  • REST dataset: This dataset, containing a larger volume of USRs with only SR labels, was used to augment training data.

The paper also mentions a "hyperparameter tuning set" (REST) for optimizing model parameters.

Key Insights and Ideas

  • Student crowdsourcing is a viable alternative to human experts: The study demonstrates that labels generated by students in a separate course can be as effective as those from dedicated human annotators for training sentiment analysis models. This is a significant finding for scalability and cost-effectiveness in educational settings.
  • Self-reported labels are unreliable for training: Students' own ratings of their reflections (SR labels) were found to be less reliable, exhibiting a left-skewed distribution and often disagreeing with the written text. This suggests that students may not accurately assess their own sentiment or may be influenced by biases.
  • Crowdsourced labels significantly improve negative sentiment detection: A key benefit of using crowdsourced labels was the substantial improvement in identifying negative sentiment (negative class recall). This is crucial for identifying students who are struggling and need support.
  • Crowdsourcing as an educational tool: The paper highlights the dual benefit of the crowdsourcing experiment, serving not only to collect data but also as a hands-on learning activity for students in an introductory data science course, teaching them about data collection, labeling, and sentiment analysis.

Key Results

  • Performance Comparison: Models trained on crowdsourced (CS) or combined (CS+SR) labels consistently outperformed models trained on self-reported (SR) labels across various metrics (weighted recall, weighted precision, weighted F1-score, and negative class recall).
  • Human Expert vs. Crowdsourced: While human expert (HE) labels generally yielded the best results, CS and CS+SR labels achieved comparable performance, particularly in detecting negative sentiment. For instance, negative class recall for RF with CS+SR labels reached 84.1%, significantly higher than with SR labels (15.4% for SVM).
  • VADER Baseline: The supervised ML models trained on both HE and crowdsourced labels significantly outperformed the unsupervised VADER approach.
  • Data Augmentation: Adding the larger REST dataset (with SR labels) to the training data improved performance, especially for negative class recall, demonstrating the benefit of more data, even if it's noisier.

Limitations and Future Work

  • Ground Truth Limitations: The study relied on HE labels for the FL18 dataset, but HE labels were not available for all datasets due to cost. This could potentially affect the performance scores.
  • Inherent Bias in USRs: The authors acknowledge that the written text itself (USRs) might not be entirely accurate, as students could be dishonest or misjudge their own experiences, leading to response and sampling biases.
  • Future Work: The authors plan to:
  • Crowdsource ground truth labels from more courses to expand the dataset.
  • Investigate the optimal number of labels required per USR for effective machine learning.
  • Develop more advanced ML models, including large language models, that can learn from multiple label types simultaneously.
  • Integrate their sentiment prediction approach into a learning analytics tool and conduct a classroom study to evaluate its usefulness for instructors in triggering interventions or improving course materials.

Summary updated

BibTeX
@inproceedings{kasumba2024practical,
  author = {Robert Kasumba and Marion Neumman},
  title = {{Practical Sentiment Analysis for Education: The Power of Student Crowdsourcing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30356},
}
Figure