A Multi-aspect Analysis of Gender Bias on Online Student Evaluations

Sofia Maria Nikolakaki, Joseph Lai, Evimaria TerziarXiv 2020 · OpenReview

TL;DR

This paper dives deep into online student evaluations of professors, using a massive dataset from Rate My Professors (RMP) spanning two decades. They found that while gender is a weak predictor of overall ratings, subtle biases emerge in the language students use and the specific "tags" they apply, revealing how gender stereotypes might influence perceptions of teaching styles and personality.

What Problem Does This Paper Address?

Student evaluations are a crucial, yet often debated, tool for assessing teaching quality. However, these evaluations can be influenced by underlying biases, including gender bias. While previous studies have touched on this, they often used small, outdated datasets. This paper aims to provide the most comprehensive analysis to date, investigating how gender bias manifests in online student evaluations across different aspects (overall ratings, specific tags, and language used) and how these biases have evolved over time and across different academic disciplines and geographic locations.

Methods

The researchers employed a multi-pronged approach:

  • Data Collection: They scraped a vast dataset of over 18.5 million evaluations from RMP, covering 4841 US institutions and 860K professors from 1999 to 2019.
  • Gender Determination: Professors' gender was inferred by analyzing the ratio of male to female pronouns in their student evaluations.
  • Feature Engineering: They extracted various features, including:
  • Directly Extracted: Professor's university, department, average overall quality (OQ) rating, level of difficulty (LOD), number of evaluations, and the date of their first evaluation.
  • Engineered: * STEM vs. Non-STEM: Categorization of departments using edit distance to match STEM lists.
  • Sentiment Analysis: Extraction of positive and negative adjectives from comments using NLTK and Vader.
  • Professor Tags: Utilizing RMP's pre-defined tags (e.g., "caring," "lots of homework") as indicators of teaching style.
  • Statistical Analysis:
    • OLS Regression: Used to model the OQ rating as a function of various features (LOD, gender, department, etc.) to identify predictors and their influence. This was done across different time periods (pre/post-2016 OQ definition change), departments, and states.
  • Logistic Regression: Employed to analyze the relationship between professor gender and the use of specific tags and adjectives, revealing how gender influences the language and descriptors used in evaluations.
  • Longitudinal and Spatial Analysis: The study specifically examined how gender bias trends have changed over the 20-year period and varied across different US states.

Datasets

The primary dataset used is from Rate My Professors (RMP), comprising over 18.5 million student evaluations collected between September 1999 and November 2019. This dataset is notable for its scale and temporal coverage, allowing for longitudinal analysis.

Key Insights and Ideas

  • Subtle but Persistent Bias: While gender is a weak predictor of the overall rating, it significantly influences the language and specific descriptors used in evaluations. This suggests that biases are not always overt but can be embedded in how students perceive and articulate their experiences.
  • Stereotypical Associations: Certain tags and adjectives are strongly associated with specific genders. For instance, "caring" is linked to female professors, while "accessible outside class" and "lecture heavy" are more associated with male professors. This highlights how gender stereotypes might shape expectations of teaching styles.
  • Evolution of Bias: The study reveals that the negative correlation between female professors and student ratings has strengthened over time, particularly since the OQ metric changed in 2016. This suggests a potential shift in how gender bias is expressed or perceived.
  • Departmental Nuances: Gender bias in teaching style perception is not uniform across disciplines. Some tags that are generally neutral in overall analysis show distinct gender associations within specific departments, indicating that disciplinary context plays a role.
  • Language as a Gender Indicator: The language used in evaluations can often predict a professor's gender, even more so than overall ratings in some cases. This underscores the power of linguistic cues in revealing underlying biases.

Key Results

  • Overall Rating: Gender is a weak negative predictor of the overall quality (OQ) rating. Female professors are slightly more likely to receive lower ratings, but the effect is small. This negative correlation has strengthened in recent years.
  • Professor Tags:
    • Tags like "caring" are positively associated with female professors.
  • Tags like "respected" and "hilarious" are negatively associated with female professors (implying they are more associated with male professors).
  • Teaching style tags show departmental variations: "participation matters" and "extra credit" are more associated with female professors, while "accessible outside class" and "graded by few things" are more with male professors.
  • Adjectives:
    • Gender-stereotypical adjectives like "beautiful" and "wonderful" are strongly associated with female professors.
  • Adjectives like "brilliant" and "great" are more associated with male professors.
  • Negative adjectives like "rude" and "unprofessional" are more frequent in female professor evaluations, while "arrogant" and "bad" are more associated with male professors.
  • The strength of certain adjective associations has changed over time (e.g., "wonderful" for females has decreased, "amazing" has increased).

Limitations and Future Work

  • Reviewer Gender Unknown: The paper acknowledges that the gender of the student reviewer is unknown, which could be a significant factor influencing evaluation bias. Future work could explore this if such data becomes available.
  • Causality: While the study identifies correlations, it cannot definitively establish causality. For example, it's unclear if female professors receive lower ratings because of bias, or if other factors (e.g., teaching styles, student demographics) contribute to this.
  • RMP Data Limitations: The study is limited to the data available on RMP, which may not perfectly represent all student evaluations or academic institutions.
  • Hypothesis Testing: The paper suggests that the increasing negative correlation between female professors and ratings might be due to contemporary biases or the new OQ metric, but RMP data doesn't allow for direct testing of these hypotheses.
  • Future Directions: The authors suggest further investigation into the impact of reviewer demographics, exploring departmental stereotypes in more detail, and analyzing how these biases might influence professor behavior.

Summary updated

Abstract

Institutions widely use student evaluations to assess the faculty's teaching performance, but underlying trends and biases can influence their interpretation. Using data from Rate My Professors, we conduct the largest and most recent quantitative data analysis to study questions related to the evaluation criteria that students have when they review the performance of their male and female professors. Our analysis spans data from two decades (1999-2019), thus taking into account recent changes on the website and in the perception of students, and demonstrates interesting insights related to how students perceive the teaching style and personality traits of their male and female professors. We also present the first analysis that investigates how gender bias evolves over time and changes over space. We believe that our results are interesting from a sociological viewpoint, as they investigate the role of gender in higher education by disclosing how students perceive and evaluate professors of different genders. In addition, we believe that our findings can be useful to educational institutions when considering possible biases that exist in the evaluations of their faculty.
BibTeX
@misc{nikolakaki2020multi,
  author = {Sofia Maria Nikolakaki and Joseph Lai and Evimaria Terzi},
  title = {{A Multi-aspect Analysis of Gender Bias on Online Student Evaluations}},
  year = {2020},
  eprint = {2011.01897},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.01897},
}

Loading related content...

Figure