Understanding Security Behavior of Real Users: Analysis of a Phishing Study

Mingqing Kang, Matthew Shonman, Anshul Subramanya, Haoruo Zhang, Xiang‐Yang Li, Anton DahburaProceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences 2021 · DOI

Abstract

This paper presents a set of statistical analyses on an empirical study of phishing email sorting by real online users. Participants were assigned to multitasking and/or incentive conditions in unattended web-based tasks that are the most realistic in any comparable study to date. Our three stages of analyses included logistic regression models to identify individual phishing “cues” contributing to successful classifications, statistical significance tests assessing the links between participants’ training experience and self-assessments of success to their actual performance, significance tests searching for significant demographic factors influencing task completion performance, and lastly k-means clustering based on a range of performance measures and utilizing participants’ demographic attributes. In particular, the results indicate that multitasking and incentives create complex dynamics while demographic traits and cybersecurity training can be informative predictors of user security behavior. These findings strongly support the benefits of security training and education and advocate for customized and differentiated interventions to increase users’ success of correctly identifying phishing emails.

Introduction

Most cybersecurity user studies suffer from "lab-grown" data, where participants behave differently under observation than they do in their natural environments. Furthermore, many studies rely on self-reported data rather than observing actual behavior. This study addresses these gaps by moving phishing research out of the controlled lab and into a high-fidelity, unattended web environment. The researchers sought to understand how external factors—specifically multitasking and monetary incentives—interact with internal factors like demographics and prior training to influence how people identify phishing attempts.

Methodology

The researchers recruited 177 participants via Amazon Mechanical Turk for a remote, web-based study. The task involved sorting 40 emails (20 phishing and 20 legitimate) into "keep" or "suspicious" folders using a real webmail interface (Roundcube). The study employed a 2x2 experimental design crossing two main variables: multitasking (answering quiz questions while sorting) and incentives (offering extra cash for high accuracy).

Experimental interface showing the email sorting task alongside the multitasking quiz.

The analysis followed a three-stage pipeline. First, logistic regression determined which "cues" (e.g., poor design, generic greetings) drove flagging decisions. Second, statistical significance tests correlated performance with demographics and training. Finally, k-means clustering identified distinct "personas" based on error rates, processing time, and confidence.

Key Insights and Results

The Multitasking Penalty

A striking finding is that multitasking acts as a "security equalizer." In single-tasking conditions, cybersecurity training significantly helped users, who achieved 86.4% accuracy compared to 74.1% for untrained users. However, when multitasking was introduced, the accuracy of trained users dropped to 70.4%, making them statistically indistinguishable from untrained users. Stress and split attention appear to create a performance ceiling that training cannot easily break.

Security Personas

Through k-means clustering, the authors identified three distinct types of users:

  • Overachievers: Characterized by low error rates, high confidence, and efficient processing.
  • Conservatives: "Safe but slow" users with very low False Negative Rates (rarely missing a phish) but high False Positive Rates (flagging too many legitimate emails).
  • Naive Users: "Trusting and vulnerable" users with high False Negative Rates but low False Positive Rates.
Comprehensive performance analysis across four dimensions: (a) error rates, (b) processing time, (c) confidence, and (d) demographics.

Demographic and Incentive Dynamics

Age and education were significant predictors of performance. While most participants with advanced degrees or training fell into the "Overachiever" cluster, those over the age of 45 often slipped into the "Naive" cluster when multitasking. Interestingly, monetary incentives did not necessarily improve results; instead, they may have pressured participants into "overthinking," leading to complex dynamics without a significant increase in accuracy.

Conclusion

The study's findings suggest that security training is not a universal solution, particularly in high-pressure environments. The researchers advocate for "customized and differentiated" interventions. Rather than one-size-fits-all training, security strategies should be tailored to the specific persona of the user—for instance, providing more automated support for "Naive" users or streamlining workflows for "Conservatives" to reduce false positives.

Summary updated

Abstract

This paper presents a set of statistical analyses on an empirical study of phishing email sorting by real online users. Participants were assigned to multitasking and/or incentive conditions in unattended web-based tasks that are the most realistic in any comparable study to date. Our three stages of analyses included logistic regression models to identify individual phishing “cues” contributing to successful classifications, statistical significance tests assessing the links between participants’ training experience and self-assessments of success to their actual performance, significance tests searching for significant demographic factors influencing task completion performance, and lastly k-means clustering based on a range of performance measures and utilizing participants’ demographic attributes. In particular, the results indicate that multitasking and incentives create complex dynamics while demographic traits and cybersecurity training can be informative predictors of user security behavior. These findings strongly support the benefits of security training and education and advocate for customized and differentiated interventions to increase users’ success of correctly identifying phishing emails.
BibTeX
@inproceedings{kang2021understanding,
  author = {Mingqing Kang and Matthew Shonman and Anshul Subramanya and Haoruo Zhang and Xiang‐Yang Li and Anton Dahbura},
  title = {{Understanding Security Behavior of Real Users: Analysis of a Phishing Study}},
  booktitle = {Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences},
  year = {2021},
  doi = {10.24251/hicss.2021.862},
}
Figure