On the Psychology of the Psychology Subject Pool

Hashem Sadeghiyeh, Siyu Wang, Hannah M. Kyllo, Maxwell R. Alberhasky, Shlishaa Savita, Kathryn L. Kellohen, Robert C. Wilson

TL;DR

This study investigates whether psychology subject pool participants' characteristics change throughout the semester, potentially biasing research findings. While some traits associated with the "good student effect" (e.g., conscientiousness, grit) are indeed more prevalent in early participants, most personality and cognitive measures show little to no correlation with participation time, suggesting that the "good student effect" might be less impactful than often assumed for many research areas.

What Problem Does This Paper Address?

Psychology research frequently relies on student participants from university subject pools, who often have the flexibility to choose when they participate. This "degree of freedom" raises a concern: do the participants who sign up early in the semester differ systematically from those who sign up late? This phenomenon, known as the "good student effect," suggests that more engaged, conscientious students might participate earlier, potentially introducing bias into research findings. This paper aims to explore the extent of this "good student effect" by examining how various demographic, personality, cognitive, decision-making, and social traits correlate with the time of semester and time of day participants enroll in studies.

Methods

The researchers conducted an exploratory analysis using data from 485 participants who completed a battery of questionnaires as part of various experiments. They measured time-of-semester by counting the days from the start of the academic term and time-of-day by converting participation time into a linear 0-1 interval. Pearson correlations were used to assess the relationships between these time variables and a wide range of individual difference measures, including demographics (age, gender, SES), Big Five personality traits, grit, optimism, need for cognition, sensation seeking, impulsivity, risk tolerance, and various social traits. To account for potential confounds, they also performed partial correlations controlling for gender. Additionally, an ANOVA was used to explore non-linear relationships between time-of-semester and personality traits.

Datasets

The study utilized data collected from 485 participants in the psychology subject pool at the University of Arizona between 2015 and 2016. The participants responded to a bundle of questionnaires administered via Qualtrics. The paper details the specific scales used for each category of individual differences measured, including established instruments like the IPIP-NEO-120 for Big Five personality traits, the Grit Scale, the Arnett Inventory of Sensation Seeking (AISS), and the Barratt Impulsivity Scale (BIS-11). No new datasets were introduced.

Key Insights and Ideas

  • The "Good Student Effect" is Real, but Nuanced: The study confirms that early participants tend to exhibit traits associated with being a "good student," such as higher conscientiousness and grit. This aligns with previous research and suggests that some degree of selection bias based on engagement does occur.
  • Sensation Seeking is an Exception: Unlike many other traits that show a linear relationship (or no relationship) with time-of-semester, Sensation Seeking exhibits a non-linear pattern, peaking in the middle and at the end of the semester. This suggests that participants seeking novelty or excitement might not follow the typical "good student" pattern.
  • Most Traits are Surprisingly Robust: The most striking finding is that the vast majority of the measured traits (personality, cognition, decision-making, social interaction) showed no significant correlation with time-of-semester or time-of-day, even before stringent corrections for multiple comparisons. This implies that for many research questions, the potential bias introduced by the "good student effect" might be minimal.
  • Time-of-Day Matters Less: Correlations between time-of-day and personality traits were generally weaker and less consistent than those for time-of-semester, suggesting it's a less significant factor in participant selection bias.

Key Results

  • Significant Correlations with Time-of-Semester:
    • Positive: Personal Distress (IRI), Sensation Seeking (total and Novelty sub-scale), and some demographic factors like higher SES and family income were associated with later participation.
  • Negative: Grit Scale, Achievement Striving (Big Five), Cooperation (Big Five), Empathic Concern (IRI), and several other Big Five subscales (e.g., Agreeableness, Openness) were associated with earlier participation.
  • Non-linear Sensation Seeking: Sensation Seeking showed a pattern of increasing from the beginning of the semester, peaking around the middle, decreasing, and then increasing again towards the end.
  • Limited Time-of-Day Effects: Only a few traits showed significant correlations with time-of-day, and these generally did not hold up after Bonferroni correction.
  • Gender Differences: Females and younger participants tended to participate earlier in the semester and earlier in the day, consistent with prior research.

Limitations and Future Work

The authors acknowledge that their study is exploratory and that the "good student effect" might be more pronounced in other subject pools or institutions with different participant demographics or recruitment strategies. They also note that while most correlations were small, for specific constructs like Grit and Sensation Seeking, researchers might need to be more mindful of recruitment timing. Future work could involve investigating these effects in different subject pools, exploring the impact of specific recruitment methods, and further examining the non-linear patterns observed for traits like Sensation Seeking. The paper also suggests that for studies focusing on traits where the "good student effect" is known to be significant, researchers might consider recruiting participants more evenly throughout the semester.

Summary updated

Abstract

Many psychology researches are performed through “psychology subject pools” which give participants considerable flexibility when they participate. This “participant degree-of-freedom” has led to concern that the characteristics of subject pool participants may change with time, with the most engaged students signing up at the start of the semester and the least engaged students leaving it all to the end. In this paper, we performed an exploratory analysis to look for evidence of this “good student effect.” Consistent with previous work, we find support for the good student effect with earlier participants scoring higher on the Big-Five subscales of Achievement-Striving and Cooperation, as well as Grit and Empathic-Concern. In addition, we found a non-linear effect of time-of-semester on Sensation-Seeking, with this measure peaking in the middle of the semester as well as the end. However, the vast majority of the measures we tested, including measures of personality, cognition, decision-making, and social interaction, did not correlate with time-of-semester or time-of-day at all. Thus, we conclude that, while some studies directly related to measures of Grit and Sensation-Seeking would do well to recruit throughout the semester, in most cases any bias introduced by the good student effect is likely to be small.
BibTeX
@article{sadeghiyeh2020psychology,
  author = {Hashem Sadeghiyeh and Siyu Wang and Hannah M. Kyllo and Maxwell R. Alberhasky and Shlishaa Savita and Kathryn L. Kellohen and Robert C. Wilson},
  title = {{On the Psychology of the Psychology Subject Pool}},
  journal = {Journal of Individual Differences},
  year = {2020},
  doi = {10.1027/1614-0001/a000327},
}

Loading related content...

Figure