Abstract
Observational data enables large-scale vaccine safety surveillance but requires careful evaluation of potential sources of bias. One potential source of bias is an index date selection procedure for the unvaccinated cohort or unvaccinated comparison time. Here, we evaluate different index date selection procedures for two vaccines: COVID-19 and influenza. For each vaccine, we extracted patient baseline characteristics on the index date and up to 450 days prior and then compared them to the characteristics of the unvaccinated patients indexed on an arbitrary date or indexed on a date of a visit. Additionally, we compared vaccinated patients indexed on the date of vaccination and the same patients indexed on a prior date or visit. COVID-19 vaccination and influenza vaccination differ drastically from each other in terms of populations vaccinated and their status on the day of vaccination. When compared to indexing on a visit in unvaccinated population, influenza vaccination had markedly higher covariate proportions and COVID-19 vaccination had lower proportions of most covariates on the index date. In contrast, COVID-19 vaccination had similar covariate proportions when compared to an arbitrary date. These effects attenuated but were still present with a longer lookback period. The effect of day 0 was present even when patients served as their own controls. Patient baseline characteristics are sensitive to the choice of the index date. In vaccine safety studies, unexposed index event should represent vaccination settings. Study designs previously used to assess influenza vaccination must be reassessed for COVID-19 to account for a potentially healthier population and lack of medical activity on the day of vaccination.
Introduction
To determine if a vaccine causes a specific side effect, researchers use large-scale electronic health records (EHR) to compare vaccinated individuals to a control group. A critical decision in these studies is choosing the "index date": the specific day you start looking for outcomes in the unvaccinated group. This process, known as anchoring, can inadvertently introduce significant bias if the healthcare context of the vaccination event is not properly matched in the control group.
Historically, influenza vaccines have been the primary model for these studies. However, COVID-19 vaccines were often delivered in non-traditional settings (like mass vaccination sites) compared to influenza vaccines, which are typically given during standard doctor visits. This paper investigates how these "anchoring effects" differ between influenza and COVID-19 vaccinations and the risks of reusing old influenza study designs for COVID-19 research.
Methods
The researchers mimicked two common observational study designs across two different vaccines (Influenza 2017–2018 and COVID-19 2020–2021):
- Cohort Design: Comparing vaccinated patients to unvaccinated patients. The unvaccinated group was indexed either on an arbitrary date (matched to a vaccination date) or on a visit date (a day they actually saw a provider).
- Self-Controlled Design: Comparing the same patient to themselves at a different time (e.g., the day of vaccination vs. a prior arbitrary date or visit).
The study utilized two massive EHR data sources mapped to the OMOP Common Data Model: Columbia University Medical Center (CUMC) and Optum EHR. They analyzed "covariate proportions"—the frequency of recorded medical conditions, procedures, and drug prescriptions—across five lookback windows ranging from "Day 0" (the index date) to 450 days prior.
Key Results
The study revealed a fundamental difference in how vaccination events are recorded. Influenza vaccination is "coupled" with healthcare utilization; it often happens because a patient is already at a clinic. In contrast, COVID-19 vaccination was largely "decoupled," occurring in pharmacies or pop-up clinics that do not always record a traditional "visit" in the EHR.
- The "Day 0" Distortion: On the day of vaccination (Day 0), 55% of influenza vaccinees had a healthcare visit recorded, compared to only 2.7% of COVID-19 vaccinees. Consequently, influenza patients showed a massive spike in medical records (blood tests, vitals, etc.) on Day 0 that was absent in the COVID-19 cohort.
- Healthy User Effect: COVID-19 vaccine recipients appeared healthier than the general population in the EHR. When compared to a "visit" in the unvaccinated population, COVID-19 vaccinees had significantly lower rates of chronic and acute diagnoses on the index date.
- Lookback Attenuation: While these differences were most extreme on Day 0, the bias persisted even when looking back a full year, suggesting that the populations being compared have fundamentally different baseline health-seeking behaviors.
Conclusion
Patient baseline characteristics are highly sensitive to the choice of the index date. For influenza, a "visit date" is often a better counterfactual for the control group because it matches the healthcare activity associated with the vaccination. For COVID-19, an "arbitrary date" is actually more accurate because it better reflects the lack of medical activity on the day of vaccination.
Study designs previously used to assess influenza vaccination must be reassessed for COVID-19 to account for a potentially healthier population and the different delivery infrastructure. Future work must focus on developing robust propensity score models that account for these delivery contexts to avoid "healthcare utilization" biases in safety surveillance.
Summary updated