Abstract
Background. COVID-19 research has relied heavily on convenience-based samples, which—though often necessary—are susceptible to important sampling biases. We begin with a theoretical overview and introduction to the dynamics that underlie sampling bias. We then empirically examine sampling bias in online COVID-19 surveys and evaluate the degree to which common statistical adjustments for demographic covariates successfully attenuate such bias. Methods. We analysed responses to identical questions from three convenience and three largely representative samples (total N = 13,731) collected online in Canada within the International COVID-19 Awareness and Responses Evaluation Study (www.icarestudy.com). We compared samples on 11 behavioural and psychological outcomes across three time points and employed multiverse-style analyses to examine how 512 combinations of demographic covariates impacted sampling discrepancies on these outcomes. Results. Significant discrepancies emerged between samples on 73% of outcomes. Participants in the convenience samples held more positive thoughts towards and engaged in more COVID-19 prevention behaviours. Covariates attenuated sampling differences in only 55% of cases and increased differences in 45%. No covariate performed reliably well. Conclusion. Our results suggest that online convenience samples may display more positive dispositions towards COVID-19 prevention behaviours being studied than would samples drawn using more representative means. Adjusting results for demographic covariates frequently increased rather than decreased bias, suggesting that researchers should be cautious when interpreting adjusted findings. Using multiverse-style analyses as extended sensitivity analyses is recommended.
The Convenience Sample Dilemma
During the COVID-19 pandemic, the urgent need for data led to a massive reliance on convenience samples—recruiting participants via social media ads or snowball sampling. While efficient, these samples are notoriously biased and rarely represent the general population.
The standard academic "fix" for this is to include demographic covariates (like age, sex, and education) in statistical models to "adjust" for these biases. However, this practice is often atheoretical; researchers include these variables because it is standard practice, not necessarily because they have a causal reason to do so. This study investigates whether this common practice actually works or if it inadvertently moves the goalposts further from the truth.
Theoretical Framework: Selection as a Collider
To understand why standard adjustments often fail, we must look at the causal dynamics of study participation. Sampling bias occurs when the factors influencing whether someone participates in a study are also related to the variables being measured.
As shown in Figure 1, study participation (Selection) acts as a "collider"—a variable influenced by both the exposure (e.g., Sex) and the outcome (e.g., Vaccine Acceptance). When an analysis is restricted only to those who chose to participate, it creates spurious associations. If a researcher adjusts for the wrong covariate—such as a mediator or another collider—they can inadvertently amplify this bias rather than reduce it.
Methodology: The Multiverse Approach
To test the reliability of demographic adjustments, the authors employed Multiverse Analysis (specifically using Specification Curves). Rather than reporting a single "best" model, they ran every possible combination of nine common demographic covariates across 11 different outcomes at three different time points.
Using data from the iCARE (International COVID-19 Awareness and Responses Evaluation) Study, the team compared six Canadian samples (Total N = 13,731):
- Three Convenience Samples: Unpaid volunteers recruited via online ads.
- Three Representative Samples: Paid participants from a professional web panel designed to match the Canadian census.
In total, they computed 16,896 unique models to quantify how often adding a covariate reduced the gap (attenuated bias) or widened it (increased bias) between the convenience and representative samples.
Key Findings: When Adjustments Backfire
The study revealed several critical insights into the nature of sampling bias in pandemic research:
- The "Pro-Prevention" Volunteer Bias: People who volunteer for COVID-19 surveys are fundamentally different from the general population. They are significantly more concerned about the virus, more likely to follow social distancing rules, and more likely to intend to get vaccinated. Initial discrepancies existed in 73% of the outcomes measured.
- Adjustments are a Double-Edged Sword: Covariates reduced the discrepancy between sample types in only 55% of cases. In the remaining 45%, the "adjustment" actually made the convenience sample less like the representative population.
- No "Silver Bullet" Covariate: No single demographic variable performed reliably well across all outcomes. While income was generally the most helpful (reducing bias in 73% of cases), variables like student status frequently exacerbated the bias.
- Magnitude of Bias: The average discrepancy was $d = .21$, a value often larger than the actual effects researchers were attempting to measure.
Conclusion: Moving Beyond Standard Fixes
The results suggest that standard demographic adjustments are surprisingly unreliable, acting more like a coin flip than a surgical tool. Adjusting results for demographics frequently increased rather than decreased bias, particularly for outcomes like hand washing where fewer than 25% of adjusted models successfully reduced bias.
For researchers using convenience samples, the study recommends moving away from "standard" atheoretical adjustments. Instead, researchers should use multiverse analyses as a sensitivity check to determine how much their conclusions shift depending on which variables are controlled for. Understanding the causal pathways—and the risks of collider bias—is essential for producing reliable insights from non-representative data.
Summary updated