Potential pitfalls in the use of real-world data for studying long COVID

Harrison G. Zhang, Jacqueline Honerlaw, Monika Maripuri, Malarkodi Jebathilagam Samayamuthu, Brendin R. Beaulieu‐Jones, Huma S. Baig, Sehi L’Yi, Yuk‐Lam Ho, Michele Morris, Vidul Ayakulangara Panickan, Xuan Wang, Griffin M. Weber, Katherine P. Liao, Shyam Visweswaran, Bryce W. Q. Tan, William Yuan, Nils Gehlenborg, Sumitra Muralidhar, Rachel Ramoni, James R. Aaron, Giuseppe Agapito, Adem Albayrak, Giuseppe Albi, M Alessiani, Anna Alloni, Danilo F. Amendola, François Angoulvant, Li L.l.j. Anthony, Bruce J. Aronow, Fatima Ashraf, Andrew M. Atz, Paul Avillach, Paula S. Azevedo, James Balshi, Brett K. Beaulieu‐Jones, Douglas S. Bell, Antonio Bellasi, Riccardo Bellazzi, Vincent Benoît, Michele Beraghi, José Luis Bernal-Sobrino, Mélodie Bernaux, Romain Bey, Surbhi Bhatnagar, Alvar Blanco-Martínez, Clara-Lea Bonzel, John Booth, Silvano Bòsari, Florence T. Bourgeois, Robert L. Bradford, Stéphane Breant, Nicholas W. Brown, Raffaele Bruno, William Bryant, Mauro Bucalo, Emily M. Bucholz, Anita Burgun, Tianxi Cai, Mario Cannataro, Aldo Carmona, Charlotte Caucheteux, Julien Champ, Jin Chen, Krista Y. Chen, Luca Chiovato, Lorenzo Chiudinelli, Kelly Cho, James J. Cimino, Tiago K. Colicchio, Sylvie Cormont, Sébastien Cossin, Jean B. Craig, Juan Luis Cruz-Bermúdez, Jaime Cruz‐Rojo, Arianna Dagliati, Mohamad Daniar, Christel Daniel, Priyam Das, Batsal Devkota, Audrey Dionne, Rui Duan, Julien Dubiel, Scott L. DuVall, Loïc Estève, Hossein Estiri, Shirley Fan, Robert W Follett, Thomas Ganslandt, Noelia García Barrio, Lana X. Garmire, Emily Getzen, Alon Geva, Tobias Gradinger, Alexandre Gramfort, Romain Griffier, Nicolas Griffon, Olivier Grisel, Alba Gutiérrez‐Sacristán, Larry Han, David A. Hanauer, Jian KangNature Medicine 2023 · DOI · OpenAlex

Executive Summary

While large-scale Electronic Health Records (EHRs) offer a significant opportunity for studying long COVID, relying solely on diagnostic codes like ICD-10 (U09.9) is often inaccurate. This study demonstrates that fewer than half of patients assigned a long-COVID code actually meet clinical definitions. To capture the true complexity of the syndrome, researchers must leverage Natural Language Processing (NLP) and machine learning to analyze clinical narratives rather than relying on structured data alone.

Introduction: The Complexity of Long COVID

Researchers are increasingly utilizing "real-world data"—the massive datasets generated by everyday healthcare interactions—to understand post-acute sequelae of SARS-CoV-2 (PASC), commonly known as long COVID. However, long COVID is not a simple, single-organ disease; it is a "constellation of symptoms" ranging from cognitive impairment ("brain fog") to extreme fatigue.

The study identifies three primary hurdles in using EHR data for this purpose:

  1. Coding Ambiguity: The diagnostic code for long COVID (U09.9) is used inconsistently by clinicians.
  2. Symptom Silencing: Standardized codes often fail to capture non-specific symptoms like fatigue, which are frequently documented in text-based clinician notes rather than structured checkboxes.
  3. Systemic Bias: Data density varies significantly; patients visiting specialized long-COVID clinics have much more detailed records than those seeing general practitioners, creating an artificial divide in how the disease is represented in datasets.

Methodology

The study was conducted by the 4CE Consortium, an international group specializing in EHR data. To validate the reliability of real-world data, the researchers performed a manual medical-record review—the clinical "gold standard"—of 300 randomly sampled individuals across three major US health systems: Beth Israel Deaconess Medical Center, University of Pittsburgh Medical Center (UPMC), and the US Veterans Health Administration (VA).

These manual reviews were compared against two major operational definitions:

  • WHO Definition: At least two new-onset persistent symptoms lasting for 60 days post-infection.
  • CDC Definition: At least one new-onset persistent symptom lasting for 30 days post-infection.

The researchers also evaluated the effectiveness of simple ICD-10 code searches against Natural Language Processing (NLP) of clinical narratives, such as discharge summaries and physician notes, to determine which method better identified the syndrome.

Key Insights: Beyond the Diagnostic Code

The ICD-10 Code as a Weak Proxy

The study found that the presence of the U09.9 code is an "unreliable surrogate" for research. Simply having the code does not guarantee that a patient meets clinical standards for long COVID.

The "Hospitalization Trap"

Approximately 12.3% of patients with the long-COVID code had previously received intensive care. The physical and psychological toll of a long ICU stay can mimic long-COVID symptoms, leading to potential misattribution where the virus itself is blamed for symptoms caused by the severity of the initial acute illness.

The Importance of Narrative Data

Symptoms such as "brain fog" are rarely captured in structured codes but are frequently mentioned in clinical notes. Incorporating narrative data via NLP substantially improved the identification of symptoms compared to using diagnosis codes alone.

Data Density Bias

Patients at specialty long-COVID clinics often appear "sicker" in the data, not necessarily due to symptom severity, but because specialists document the syndrome more thoroughly. This "granularity gap" can confound research findings if not properly adjusted for in statistical models.

Quantitative Results

The discrepancy between coding and clinical reality is stark:

  • Low Clinical Agreement: Among patients assigned the long-COVID ICD-10 code, only 40.2% met the strict WHO definition, and 65.4% met the broader CDC definition.
  • The Specialty Gap: Only 24.0% of patients with a long-COVID code actually visited a specialty long-COVID clinic. Furthermore, of those who met the WHO definition, only 35.6% were seen at these specialty facilities.
  • Under-reporting: 58.3% of patients with the U09.9 code had only a single symptom that met the WHO criteria, suggesting the code is often used as a catch-all for any lingering post-viral issue.

Conclusion and Future Directions

Defining the onset of long COVID is becoming increasingly difficult due to the prevalence of at-home testing, which is rarely recorded in EHRs. The authors suggest that researchers move toward using "infection time periods" rather than specific index dates.

The paper concludes by advocating for the development of "computational phenotypes." Instead of relying on a single diagnostic code, researchers should employ semi-supervised machine-learning algorithms to identify complex temporal patterns in the data. These models are better equipped to distinguish between pre-existing conditions and new-onset symptoms, leading to a more equitable and accurate understanding of long COVID.

Summary updated

BibTeX
@inproceedings{zhang2023potential,
  author = {Harrison G. Zhang and Jacqueline Honerlaw and Monika Maripuri and Malarkodi Jebathilagam Samayamuthu and Brendin R. Beaulieu‐Jones and Huma S. Baig and Sehi L’Yi and Yuk‐Lam Ho and Michele Morris and Vidul Ayakulangara Panickan and Xuan Wang and Griffin M. Weber and Katherine P. Liao and Shyam Visweswaran and Bryce W. Q. Tan and William Yuan and Nils Gehlenborg and Sumitra Muralidhar and Rachel Ramoni and James R. Aaron and Giuseppe Agapito and Adem Albayrak and Giuseppe Albi and M Alessiani and Anna Alloni and Danilo F. Amendola and François Angoulvant and Li L.L.J. Anthony and Bruce J. Aronow and Fatima Ashraf and Andrew M. Atz and Paul Avillach and Paula S. Azevedo and James Balshi and Brett K. Beaulieu‐Jones and Douglas S. Bell and Antonio Bellasi and Riccardo Bellazzi and Vincent Benoît and Michele Beraghi and José Luis Bernal-Sobrino and Mélodie Bernaux and Romain Bey and Surbhi Bhatnagar and Alvar Blanco-Martínez and Clara-Lea Bonzel and John Booth and Silvano Bòsari and Florence T. Bourgeois and Robert L. Bradford and Stéphane Breant and Nicholas W. Brown and Raffaele Bruno and William Bryant and Mauro Bucalo and Emily M. Bucholz and Anita Burgun and Tianxi Cai and Mario Cannataro and Aldo Carmona and Charlotte Caucheteux and Julien Champ and Jin Chen and Krista Y. Chen and Luca Chiovato and Lorenzo Chiudinelli and Kelly Cho and James J. Cimino and Tiago K. Colicchio and Sylvie Cormont and Sébastien Cossin and Jean B. Craig and Juan Luis Cruz-Bermúdez and Jaime Cruz‐Rojo and Arianna Dagliati and Mohamad Daniar and Christel Daniel and Priyam Das and Batsal Devkota and Audrey Dionne and Rui Duan and Julien Dubiel and Scott L. DuVall and Loïc Estève and Hossein Estiri and Shirley Fan and Robert W Follett and Thomas Ganslandt and Noelia García Barrio and Lana X. Garmire and Emily Getzen and Alon Geva and Tobias Gradinger and Alexandre Gramfort and Romain Griffier and Nicolas Griffon and Olivier Grisel and Alba Gutiérrez‐Sacristán and Larry Han and David A. Hanauer and Jian Kang},
  title = {{Potential pitfalls in the use of real-world data for studying long COVID}},
  booktitle = {Nature Medicine},
  year = {2023},
  doi = {10.1038/s41591-023-02274-y},
}
Figure