How well do covariates perform when adjusting for sampling bias in online COVID-19 research? Insights from multiverse analyses

Keven Joyal‐Desmarais, Jovana Stojanovic, Eric B. Kennedy, Joanne Enticott, Vincent Gosselin Boucher, Hung Vo, Urška Košir, Kim Lavoie, Simon Bacon, Zahir Vally, Nora Grañana, Analía Verónica Losada, Jacqueline Boyle, Md Shajedur Rahman Shawon, Shrinkhala Dawadi, Helena Teede, Alexandra Kautzky‐Willer, Arobindu Dash, Marília Estêvam Cornélio, Marlus Karsten, Darlan Laurício Matte, Felipe Fossati Reichert, Ahmed M Abou-Setta, Shawn D. Aaron, Angela S. Alberga, Tracie A. Barnett, Silvana Barone, Ariane Bélanger‐Gravel, Sarah Bernard, Lisa Maureen Birch, Susan J. Bondy, Linda Booij, Roxane Borgès Da Silva, Jean Bourbeau, Rachel Burns, Tavis S. Campbell, Linda E. Carlson, Étienne Charbonneau, Kim Corace, Olivier Drouin, Francine M. Ducharme, Mohsen Farhadloo, Carl F. Falk, Richard Fleet, Michel Fournier, Gary Garber, Lise Gauvin, Jennifer Gordon, Roland Grad, Samir Gupta, Kim Hellemans, Catherine M. Herba, Heungsun Hwang, Jack Jedwab, Lisa Kakinami, Sunmee Kim, Joanne Liu, Colleen M. Norris, Sandra Peláez, Louise Pilote, Paul Poirier, Justin Presseau, Eli Puterman, Joshua A. Rash, Paula Aver Bretanha Ribeiro, Mohsen Sadatsafavi, Paramita Saha‐Chaudhuri, Eva Suarthana, SzeMan Tse, Michael Vallis, Nicolás Bronfman Caceres, Manuel S. Ortíz, Paula Repetto, Mariantonia Lemos, Angelos P. Kassianos, Naja Hulvej Rod, Mathieu Beraneck, Grégory Ninot, Beate Ditzen, Thomas Kubiak, Sam Codjoe, Lily Kpobi, Amos Laar, Theodora Skoura, Delfin Lovelina Francis, Naorem Kiranmala Devi, Sanjenbam Meitei, Suzanne Tanya Nethan, Lancelot Pinto, Kallur Nava Saraswathy, Dheeraj Tumu, Silviana Lestari, Grace Wangge, Molly Byrne, Hannah Durand, Jenny McSharry, Oonagh Meade, Gerard J. Molloy, Chris Noone, Hagai LevineEuropean 2022 · DOI

Abstract

COVID-19 research has relied heavily on convenience-based samples, which—though often necessary—are susceptible to important sampling biases. This study analyzed responses from three convenience and three largely representative samples (total N = 13,731) in Canada. We compared samples on 11 behavioral and psychological outcomes across three time points and employed multiverse-style analyses to examine how 512 combinations of demographic covariates impacted sampling discrepancies. Significant discrepancies emerged on 73% of outcomes. Covariates attenuated sampling differences in only 55% of cases and increased differences in 45%. Our results suggest that online convenience samples may display more positive dispositions towards COVID-19 prevention behaviors, and adjusting for demographic covariates frequently increases rather than decreases bias.

The Problem: Selection and Collider Bias

During the pandemic, the speed and cost-effectiveness of online convenience samples (e.g., social media recruitment) made them a staple of behavioral research. The standard remedy for the inherent selection bias in these samples is to include demographic covariates like age, sex, and education in statistical models, assuming this "adjusts" the results toward representativeness.

However, this practice often lacks a grounding in causal theory. Without considering the underlying causal paths, "adjusting" for variables can lead to collider bias. This occurs when researchers condition on a variable that is a common effect of both the exposure and the outcome, or other factors related to study participation. As shown in the causal diagrams below, selection itself acts as a collider, potentially creating or distorting associations.

Causal diagrams illustrating selection as a collider and the impact of adjusting for various types of variables (confounders, mediators, and colliders).

Methodology: A Multiverse Approach

To evaluate the effectiveness of demographic adjustments, the researchers employed a multiverse analysis (specifically, specification curve analysis). Rather than selecting a single "best" model, they ran 512 unique regression models for every outcome at every time point. These models represented every possible combination of nine common demographic covariates: sex, age, education, income, ethnicity, province, language, mother tongue, and student status.

The study utilized data from the International COVID-19 Awareness and Responses Evaluation (iCARE) Study, comparing three convenience samples against three "gold standard" representative web-panel samples matched to Canadian census data. As shown in the demographic breakdown, convenience samples often skewed significantly from the general population, particularly in terms of sex and education levels.

Demographic comparison between the 2016 Canadian Census and the iCARE convenience and representative samples across three time points.

Key Insights: The "Pro-Prevention" Bias

A critical finding is that participants in convenience samples are psychologically distinct from the general population. They exhibit a "selective exposure" effect, being more concerned about the virus and more likely to adhere to health mandates. For instance, convenience samples showed vaccine intentions that were 13–18% higher than representative samples. This suggests that those who volunteer for COVID-19 studies are often those already most engaged with prevention measures.

Results: The Adjustment Coin Flip

The multiverse analysis revealed that blindly adding covariates is not a neutral act:

  • Baseline Bias: Significant discrepancies existed between convenience and representative samples on 73% of outcomes.
  • Performance: Across approximately 17,000 models, covariate adjustment reduced bias in only 55% of cases. In the remaining 45% of cases, adjustment actually increased the discrepancy between the convenience sample and the representative "truth."
  • No Universal Fix: No single covariate consistently improved representativeness across all outcomes. While adjustments worked well for behaviors like "avoiding social gatherings," they almost universally worsened bias for others, such as "hand washing."

Conclusion and Recommendations

The common practice of demographic adjustment in convenience samples is essentially a coin flip. Researchers should be cautious when interpreting adjusted findings, as they may inadvertently introduce more bias than they remove.

The authors recommend moving away from "rules of thumb" for covariate selection. Instead, researchers should:

  1. Use Directed Acyclic Graphs (DAGs) to theorize causal relationships before selecting covariates.
  2. Employ multiverse-style sensitivity analyses to report how results vary across different modeling choices.
  3. Consider psychological covariates, rather than just demographics, to account for the "pro-prevention" bias inherent in health-related volunteer samples.

Summary updated

Abstract

COVID-19 research has relied heavily on convenience-based samples, which-though often necessary-are susceptible to important sampling biases. We begin with a theoretical overview and introduction to the dynamics that underlie sampling bias. We then empirically examine sampling bias in online COVID-19 surveys and evaluate the degree to which common statistical adjustments for demographic covariates successfully attenuate such bias. This registered study analysed responses to identical questions from three convenience and three largely representative samples (total N = 13,731) collected online in Canada within the International COVID-19 Awareness and Responses Evaluation Study ( www.icarestudy.com ). We compared samples on 11 behavioural and psychological outcomes (e.g., adherence to COVID-19 prevention measures, vaccine intentions) across three time points and employed multiverse-style analyses to examine how 512 combinations of demographic covariates (e.g., sex, age, education, income, ethnicity) impacted sampling discrepancies on these outcomes. Significant discrepancies emerged between samples on 73% of outcomes. Participants in the convenience samples held more positive thoughts towards and engaged in more COVID-19 prevention behaviours. Covariates attenuated sampling differences in only 55% of cases and increased differences in 45%. No covariate performed reliably well. Our results suggest that online convenience samples may display more positive dispositions towards COVID-19 prevention behaviours being studied than would samples drawn using more representative means. Adjusting results for demographic covariates frequently increased rather than decreased bias, suggesting that researchers should be cautious when interpreting adjusted findings. Using multiverse-style analyses as extended sensitivity analyses is recommended.
BibTeX
@inproceedings{joyaldesmarais2022how,
  author = {Keven Joyal‐Desmarais and Jovana Stojanovic and Eric B. Kennedy and Joanne Enticott and Vincent Gosselin Boucher and Hung Vo and Urška Košir and Kim Lavoie and Simon Bacon and Zahir Vally and Nora Grañana and Analía Verónica Losada and Jacqueline Boyle and Md Shajedur Rahman Shawon and Shrinkhala Dawadi and Helena Teede and Alexandra Kautzky‐Willer and Arobindu Dash and Marília Estêvam Cornélio and Marlus Karsten and Darlan Laurício Matte and Felipe Fossati Reichert and Ahmed M Abou-Setta and Shawn D. Aaron and Angela S. Alberga and Tracie A. Barnett and Silvana Barone and Ariane Bélanger‐Gravel and Sarah Bernard and Lisa Maureen Birch and Susan J. Bondy and Linda Booij and Roxane Borgès Da Silva and Jean Bourbeau and Rachel Burns and Tavis S. Campbell and Linda E. Carlson and Étienne Charbonneau and Kim Corace and Olivier Drouin and Francine M. Ducharme and Mohsen Farhadloo and Carl F. Falk and Richard Fleet and Michel Fournier and Gary Garber and Lise Gauvin and Jennifer Gordon and Roland Grad and Samir Gupta and Kim Hellemans and Catherine M. Herba and Heungsun Hwang and Jack Jedwab and Lisa Kakinami and Sunmee Kim and Joanne Liu and Colleen M. Norris and Sandra Peláez and Louise Pilote and Paul Poirier and Justin Presseau and Eli Puterman and Joshua A. Rash and Paula Aver Bretanha Ribeiro and Mohsen Sadatsafavi and Paramita Saha‐Chaudhuri and Eva Suarthana and SzeMan Tse and Michael Vallis and Nicolás Bronfman Caceres and Manuel S. Ortíz and Paula Repetto and Mariantonia Lemos and Angelos P. Kassianos and Naja Hulvej Rod and Mathieu Beraneck and Grégory Ninot and Beate Ditzen and Thomas Kubiak and Sam Codjoe and Lily Kpobi and Amos Laar and Theodora Skoura and Delfin Lovelina Francis and Naorem Kiranmala Devi and Sanjenbam Meitei and Suzanne Tanya Nethan and Lancelot Pinto and Kallur Nava Saraswathy and Dheeraj Tumu and Silviana Lestari and Grace Wangge and Molly Byrne and Hannah Durand and Jenny McSharry and Oonagh Meade and Gerard J. Molloy and Chris Noone and Hagai Levine},
  title = {{How well do covariates perform when adjusting for sampling bias in online COVID-19 research? Insights from multiverse analyses}},
  booktitle = {European},
  year = {2022},
  doi = {10.1007/s10654-022-00932-y},
}
Figure