Analyzing Gender Bias in Student Evaluations

Andamlak Terkik, Emily Prud’hommeaux, Cecilia Ovesdotter Alm, Christopher M. Homan, Scott FranklinCOLING 2016

TL;DR

This paper investigates gender bias in student evaluations of university instructors by analyzing open-ended text comments, moving beyond traditional numerical ratings. While finding no significant gender bias in overall satisfaction scores, the study reveals that students use distinctly different language to describe male and female instructors, suggesting subtle, qualitative biases persist.

What Problem Does This Paper Address?

Student evaluations of teaching (SETs) are a cornerstone of academic assessment, influencing tenure, promotion, and compensation. However, a growing body of research indicates that these evaluations can be subject to gender bias, with female instructors and instructors of color often receiving lower numerical scores, regardless of actual teaching quality. Previous studies have primarily focused on numerical ratings, leaving the nuances of qualitative feedback in open-ended comments largely unexplored. This paper aims to bridge that gap by computationally analyzing the language used in student comments to uncover potential gender-based differences in how instructors are described and perceived.

Methods

The researchers employed a two-tier machine learning approach for analyzing student comments.

  • Sentiment Analysis: The first tier involved training a supervised classifier to determine the overall sentiment (satisfaction level) of each comment. This classifier utilized two types of lexical features:
  • N-gram features: Unigrams, bigrams, and trigrams were extracted to capture local contextual information.
  • Sentiment term scores: Aggregate positive and negative scores were computed for each comment using three established sentiment lexicons (MPQA, NRC emotion, and Bing Liu's opinion lexicons).
  • Stacked Generalization: The predictions from four models (random forest, multinomial naive Bayes, Bernoulli naive Bayes, and logistic regression) in the first tier, along with the sentiment term scores, were used to train a final classifier in the second tier, also a random forest.
  • Gendered Language Analysis: To explore differences in language use, the researchers normalized and lemmatized comments and then ranked words based on their mutual information (MI) with each gender. They categorized these words into those referring to the instructor and those describing the instructor or student experience.

Datasets

The study utilized three datasets of student evaluation comments:

  • small-labeled: 2,076 comments from two introductory statistics courses, manually labeled with five satisfaction levels (very satisfied to very dissatisfied). This dataset was used for training and testing the satisfaction classifier.
  • medium-unlabeled: 15,896 comments from various STEM courses, used for predicting satisfaction levels with the trained classifier.
  • large-unlabeled: 107,855 comments from various STEM courses, used for computing analytics and identifying terms strongly associated with instructor gender.

Key Insights and Ideas

  • Satisfaction is Gender-Neutral (Quantitatively): Contrary to some previous findings based on numerical ratings, this study found that the overall satisfaction level expressed in written comments does not appear to be significantly influenced by the instructor's gender. This suggests that while numerical scores might show bias, the qualitative sentiment in open-ended feedback might be more equitable.
  • Language is Gendered (Qualitatively): The most striking finding is that students use demonstrably different language when describing male and female instructors, even when the overall sentiment is similar. This points to subtle, qualitative biases that are not captured by simple satisfaction scores.
  • Titles and Address Matter: Students are more likely to use formal titles (Prof., Dr.) and refer to male instructors by their last names, while female instructors are more frequently addressed by their first names or generic terms like "teacher" or "instructor." This suggests an unwarranted bias in the use of prestigious titles and respectful forms of address.
  • Positive Descriptors Differ: Female instructors are more often described with generic positive terms like "amazing" and "wonderful," while male instructors are more frequently characterized by personal qualities like "funny," "knowledgeable," and "interesting." This could indicate a tendency to praise women for perceived personality traits rather than teaching effectiveness, and men for intellectual or engaging qualities.
  • Impact on Learning vs. Personal Qualities: Students described female instructors more in terms of how they impacted learning experiences (e.g., "organized," "willing," "helpful"). In contrast, male instructors were more often recognized for personal qualities that may be independent of their teaching ability.

Key Results

  • The ensemble classifier achieved an accuracy of 81% on the merged task (distinguishing satisfied from dissatisfied comments), outperforming a majority-class baseline and an SVM classifier.
  • For the manually labeled "small-labeled" dataset, female instructors received slightly more favorable satisfaction ratings (74% satisfied) compared to male instructors (62% satisfied). However, this difference disappeared in the "medium-unlabeled" dataset, where the classifier predicted satisfaction levels to be equally distributed (94% satisfied for both genders).
  • Analysis of the "large-unlabeled" dataset revealed significant differences in language use:
  • Terms of address: "Prof./professor" and "<Last name>" were used more for men, while "<First name>" was used more for women.
  • Descriptive words: Women were more likely to be described with terms like "amazing," "wonderful," and "loved." Men were more likely to be described with terms like "funny," "knowledgeable," and "interesting." The term "tangent" (a negative descriptor) was used more for men.

Limitations and Future Work

  • Subjectivity of Annotations: The authors acknowledge that satisfaction is a subjective concept and that manual annotations can be influenced by annotator perception. While inter-rater agreement was moderate, it highlights the inherent subjectivity.
  • Classifier Artifacts: The disappearance of gender differences in satisfaction in the larger dataset could be an artifact of the classifier, the larger dataset size, or the broader range of courses.
  • Future Directions: The researchers plan to:
  • Develop the classifier using more complex language features (e.g., semantic role labels, word embeddings).
  • Explore the relationship between numerical ratings and automatically generated satisfaction scores.
  • Investigate syntactic structures associated with gendered language.
  • Analyze confounding factors like subject matter, instructor rank, and student demographics to further understand the nuances of gender bias.

Summary updated

BibTeX
@inproceedings{terkik2016analyzing,
  author = {Andamlak Terkik and Emily Prud’hommeaux and Cecilia Ovesdotter Alm and Christopher M. Homan and Scott Franklin},
  title = {{Analyzing Gender Bias in Student Evaluations}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure