Abstract
Question: When accounted for all potential biases, what is the weekly effectiveness of COVID-19 vaccines?
Findings: In this cohort study, we replicated the results of randomized clinical trials (RCTs), discovered a plausible increase in effectiveness after week one following the first dose of mRNA COVID-19 vaccines, and identified differences in temporal trends of vaccine exposure and baseline characteristics in vaccinated groups.
Meaning: Observational data can be used to reliably estimate vaccine effectiveness if biases are accounted for. Vaccines need to be directly compared.
Introduction
While Randomized Clinical Trials (RCTs) remain the gold standard for evaluating vaccine efficacy, they are often limited by high costs, specific inclusion criteria, and finite durations. Electronic Health Records (EHRs) offer a massive, real-world alternative for monitoring vaccine performance in diverse populations. However, observational data is notoriously prone to systematic biases that can lead to contradictory results.
This study, conducted by researchers at Columbia University, investigates whether discrepancies in early COVID-19 vaccine effectiveness (VE) estimates—particularly the high protection reported immediately after the first dose—stem from flaws in the data or hidden biases in analytical methodologies. By analyzing EHR data for nearly 180,000 patients, the researchers sought to determine if sophisticated "de-biasing" techniques could align real-world evidence with clinical trial results.
Methodology
The researchers employed a retrospective cohort study design using the Columbia University Irving Medical Center (CUIMC) database, mapped to the OMOP Common Data Model. To ensure comprehensive coverage, the database was linked to New York City and State Immunization Registries, capturing vaccinations administered at pharmacies and mass-vaccination sites.
To address potential confounding factors, the study utilized several advanced statistical techniques:
- Large-Scale Propensity Score Matching (PSM): The study used up to 54,987 covariates—including age, gender, specific drug exposures, and medical history—to ensure vaccinated and unvaccinated groups were comparable.
- Anchoring Strategies: Researchers compared two methods for setting the "start date" for control groups: matching unvaccinated patients to a specific calendar date versus matching them to a specific medical visit.
- Negative Control Calibration: 93 "negative control" outcomes (conditions unrelated to the vaccine) were used to detect and correct for residual systematic bias.
- Manual Chart Review: A granular review of medical charts was conducted for patients testing positive in the first week post-vaccination to understand the clinical context of their diagnoses.
Identifying and Mitigating Bias
The study identified several critical sources of bias that frequently distort observational vaccine research:
The "Side Effect" Behavioral Bias
Manual chart reviews revealed a significant behavioral discrepancy: in the first week following vaccination, individuals were less likely to seek care for COVID-like symptoms, often attributing them to expected vaccine side effects. Conversely, unvaccinated individuals sought care immediately for similar symptoms. This difference in healthcare-seeking behavior artificially inflated the perceived effectiveness of the vaccine in the first seven days.
Incidental Findings vs. Symptomatic Disease
Vaccinated patients diagnosed in the first week were often "incidental" cases—individuals who tested positive during routine screening for unrelated procedures (e.g., surgery). In contrast, unvaccinated patients were more likely to be at the hospital specifically because they were symptomatic. Furthermore, the study found that relying on ICD-10 billing codes (like U07.1) was less accurate than confirmed lab tests, as billing codes are often used for documenting past infections or screening rather than active disease.
Date Anchoring
The researchers found that matching unvaccinated patients to a specific date rather than a medical visit provided a more balanced distribution of baseline characteristics. This approach effectively reduced "healthy user" bias, where individuals who frequently visit the doctor for preventive care (and are thus more likely to be vaccinated) skew the data.
Results
By accounting for these biases, the study successfully replicated the high efficacy rates reported in the original mRNA vaccine clinical trials.
- Weekly Trajectory: The vaccine effectiveness (VE) for Pfizer and Moderna vaccines showed a biologically plausible climb, reaching approximately 84% for infection prevention and 86% for hospitalization prevention by week six.
- Correcting the Week One Anomaly: While raw data suggested an unexpectedly high VE of 58% in the first week, the researchers demonstrated that this was a result of behavioral biases and incidental findings rather than immediate biological protection.
Conclusion
The findings demonstrate that observational EHR data is a powerful and reliable tool for estimating vaccine effectiveness, provided that researchers rigorously account for health-seeking behaviors and data definitions. While the study was limited by a small sample size for the Janssen (J&J) vaccine—making direct head-to-head comparisons difficult—it establishes a framework for using real-world data to monitor public health interventions. Future research should focus on large-scale direct comparisons to better understand comparative effectiveness across diverse populations and vaccine types.
Summary updated