A metadata framework for computational phenotypes

Henry M. Spotnitz, Nripendra Acharya, James J. Cimino, Shawn N. Murphy, Bahram Namjou, Nancy A. Crimmins, Theresa L. Walunas, Cong Liu, David R. Crosslin, Barbara Benoit, Elisabeth A. Rosenthal, Jennifer A. Pacheco, Anna Ostropolets, Harry Reyes Nieva, Jason Patterson, Lauren R. Richter, Tiffany J. Callahan, Ahmed El‐Hussein, Chao Pang, Krzysztof Kiryluk, Jordan G. Nestor, Atlas Khan, Sumit Mohan, Evan Minty, Wendy K. Chung, Wei‐Qi Wei, Karthik Natarajan, Chunhua WengJAMIA 2023 · DOI

TL;DR

This paper introduces a novel metadata framework designed to improve the discoverability and reusability of computational phenotypes. By standardizing descriptions of phenotype definitions, algorithms, and validation metrics, this framework aims to reduce redundancy and enhance the rigor of observational health research.

What Problem Does This Paper Address?

The increasing development of computational phenotypes, which are essentially code-based definitions of clinical concepts, presents a challenge for researchers. These phenotypes are crucial for large-scale precision medicine and observational studies, but their current descriptions are often inconsistent, making it difficult to find the right phenotype for a specific task. This inconsistency leads to redundant efforts in creating similar phenotypes and can result in different patient cohorts being identified even when using phenotypes with the same name (e.g., "diabetes"). The paper aims to solve this by creating a standardized way to describe these phenotypes, making them more accessible, reusable, and reliable.

Methods

The researchers employed a mixed-methods approach. First, they convened a working group of 20 phenotyping experts from two large research networks (eMERGE and OHDSI) to brainstorm and reach a consensus on essential metadata elements. This process resulted in a framework with 39 metadata elements. Second, they surveyed 47 new researchers using a 5-point Likert scale and open-ended questions to evaluate the utility of these elements. Finally, two researchers pilot-tested the framework by annotating 8 type-2 diabetes mellitus phenotypes.

Datasets

The study did not introduce new datasets for training or evaluation. Instead, it focused on developing a framework for annotating existing computational phenotypes. The pilot annotation involved 8 type-2 diabetes mellitus phenotypes, with 3 sourced from the PheKB library and 5 from a prior publication.

Key Insights and Ideas

  • The Need for Standardization is High: The sheer number of existing, often redundant, phenotypes highlights a critical need for better organization and description.
  • Metadata is Key to Reusability: Explicit and comprehensive metadata is essential for researchers to confidently select and reuse phenotypes across different studies and institutions.
  • Validation is Paramount: A significant portion of the framework's utility lies in its ability to capture detailed information about how phenotypes were validated, including methods, populations, and metrics. This builds trust and transparency.
  • User Experience Matters: While the framework is comprehensive, its complexity and the human cost of data collection are acknowledged limitations, suggesting a need for optimization.

Key Results

  • The developed framework consists of 39 metadata elements categorized into Background Information, Algorithm Specifics, Performance Metrics, and Limitations.
  • During the pilot annotation, two researchers were able to annotate 8 type-2 diabetes mellitus phenotypes within 60 minutes each, demonstrating the framework's feasibility.
  • Over 90% of survey respondents rated metadata elements related to phenotype definition and validation methods/metrics highly (scores of 4 or 5 on a 5-point Likert scale).
  • Thematic analysis revealed that the framework is effective in capturing rich and explicit descriptions, facilitating phenotype search, ensuring compliance with data standards, and detailing validation metrics.

Limitations and Future Work

The authors acknowledge that the framework's complexity can lead to significant human costs for data collection. They also note that the framework may need updates, particularly for machine-learning-based phenotypes and to better support unstructured responses. Future work could explore structuring the framework's responses more effectively (e.g., using a combination of free text, semi-structured, and unstructured formats) and potentially incorporating a data dictionary to further standardize responses. The goal is to improve the consistency and quality of computable phenotypes for better data curation and research.

Summary updated

Abstract

With the burgeoning development of computational phenotypes, it is increasingly difficult to identify the right phenotype for the right tasks. This study uses a mixed-methods approach to develop and evaluate a novel metadata framework for retrieval of and reusing computational phenotypes. Twenty active phenotyping researchers from 2 large research networks, Electronic Medical Records and Genomics and Observational Health Data Sciences and Informatics, were recruited to suggest metadata elements. Once consensus was reached on 39 metadata elements, 47 new researchers were surveyed to evaluate the utility of the metadata framework. The survey consisted of 5-Likert multiple-choice questions and open-ended questions. Two more researchers were asked to use the metadata framework to annotate 8 type-2 diabetes mellitus phenotypes. More than 90% of the survey respondents rated metadata elements regarding phenotype definition and validation methods and metrics positively with a score of 4 or 5. Both researchers completed annotation of each phenotype within 60 min. Our thematic analysis of the narrative feedback indicates that the metadata framework was effective in capturing rich and explicit descriptions and enabling the search for phenotypes, compliance with data standards, and comprehensive validation metrics. Current limitations were its complexity for data collection and the entailed human costs.
BibTeX
@inproceedings{spotnitz2023metadata,
  author = {Henry M. Spotnitz and Nripendra Acharya and James J. Cimino and Shawn N. Murphy and Bahram Namjou and Nancy A. Crimmins and Theresa L. Walunas and Cong Liu and David R. Crosslin and Barbara Benoit and Elisabeth A. Rosenthal and Jennifer A. Pacheco and Anna Ostropolets and Harry Reyes Nieva and Jason Patterson and Lauren R. Richter and Tiffany J. Callahan and Ahmed El‐Hussein and Chao Pang and Krzysztof Kiryluk and Jordan G. Nestor and Atlas Khan and Sumit Mohan and Evan Minty and Wendy K. Chung and Wei‐Qi Wei and Karthik Natarajan and Chunhua Weng},
  title = {{A metadata framework for computational phenotypes}},
  booktitle = {JAMIA},
  year = {2023},
  doi = {10.1093/jamiaopen/ooad032},
}

Loading related content...

Figure