I-ODA, Real-world Multi-modal Longitudinal Data for Ophthalmic Applications

Nooshin Mojab, Vahid Noroozi, Abdullah Aleem, Manoj Prabhakar Nallabothula, Joseph Baker, Dimitri T. Azar, Mark Rosenblatt, R. V. Paul Chan, Darvin Yi, Philip S. Yu, Joelle A. Hallak

TL;DR

This paper introduces I-ODA, a novel, large-scale, multi-modal, longitudinal ophthalmic imaging dataset collected from a real-world clinical setting. I-ODA aims to bridge the gap between AI model performance in controlled environments and their translation to clinical practice by providing a diverse and comprehensive dataset that captures the complexities of real-world patient data.

What Problem Does This Paper Address?

The field of artificial intelligence (AI) in healthcare, particularly in ophthalmology, has seen rapid advancements thanks to deep learning. However, a significant challenge remains: translating these high-performing models from research settings to real-world clinical applications. This translation is hindered by several factors, including the lack of adequate, diverse, and longitudinal patient data. Existing public datasets often suffer from limited patient numbers, artificial settings (like clinical trials), a lack of longitudinal information, and potential biases due to limited diversity. This makes it difficult to develop AI algorithms that generalize well and are robust enough for clinical use.

Methods

The authors developed a comprehensive infrastructure to collect, preprocess, annotate, and anonymize ophthalmic imaging data. Key methodological contributions include:

  • Multi-modal Data Integration: Combining data from multiple sources: an in-house server for raw images, a SQL database for patient metadata and imaging sessions, and a hospital billing system for diagnoses and interventions.
  • Modality Tagging Pipeline: A two-step process to automatically assign image modalities. First, a similarity-based classification using a pretrained ResNet-50 extracts features and compares them to prototype images. Second, this tag is verified using imaging device information.
  • Disease Annotation: Integrating patient metadata, diagnoses, and interventions from the SQL and billing systems to create a rich, annotated dataset.
  • Data Anonymization: A rigorous process to protect patient privacy, adhering to HIPAA regulations. This involved image anonymization using K-means clustering to group images by visual characteristics and then applying category-specific masking, and metadata anonymization by replacing identifiers and preserving longitudinal information through age and relative date calculations.

Datasets

The paper introduces the Illinois Ophthalmic Database Atlas (I-ODA).

  • Size: Contains 3,668,649 ophthalmic images from 33,876 individuals, collected over 12 years.
  • Modalities: Encompasses 12 different imaging modalities, including OCT, Fundus, HVF, B-Scans, Corneal Topography, and various report types.
  • Characteristics:
    • Real-world: Collected from a single academic medical center (Illinois Eye and Ear Infirmary).
  • Longitudinal: Captures patient data over multiple visits.
  • Multi-modal: Includes various imaging types.
  • Multi-domain: Data generated by over 15 different imaging devices.
  • Diverse Patient Population: Includes demographic information (age, gender, ethnicity, race, location).
  • Rich Annotations: Includes diagnoses and interventions.

Key Insights and Ideas

  • The Critical Need for Real-World Data: The paper strongly emphasizes that AI models trained on artificial or limited datasets struggle to generalize to the complexities and variability of real-world clinical data. I-ODA is designed to address this directly.
  • Longitudinal Data is Key for Understanding Disease Progression: The dataset's longitudinal nature is crucial for studying how diseases evolve over time, which is essential for developing predictive and prognostic AI models.
  • Multi-modal Data Unlocks Deeper Insights: Combining different imaging modalities provides a more comprehensive view of the eye, allowing for richer analysis and potentially more accurate diagnoses and predictions.
  • Infrastructure for Data Curation is as Important as the Data Itself: The paper details the significant effort involved in building the infrastructure for data collection, annotation, and anonymization, highlighting the challenges and solutions for creating such a valuable resource.

Key Results

The paper primarily focuses on introducing and describing the I-ODA dataset. While quantitative results are not the main focus, the authors demonstrate the dataset's utility through two example applications:

  • Glaucoma Detection: Using I-ODA, they formulated a multi-task framework for glaucoma detection and segmentation, achieving a 2.6% improvement in cup segmentation over baselines by leveraging the dataset's rich data.
  • Generalization Studies: They used I-ODA in a transfer learning framework to show that models trained on complex, real-world data generalize better to standardized datasets, achieving a 16% relative improvement.

Limitations and Future Work

  • Limitations:
    • Annotation Complexity: The authors acknowledge that annotations, while comprehensive, are derived from retrospective data and may have inherent limitations or noise.
  • Single Institution: While valuable, the data originates from a single academic medical center, which might limit generalizability to other healthcare systems with different patient demographics or clinical practices.
  • Data Release Pending Legal Approval: The dataset is not yet publicly available.
  • Future Work:
    • Continued Growth: The I-ODA databank is intended to grow in imaging and patient metadata.
  • Validation of Algorithms: The dataset will serve as a benchmark for validating current algorithms and developing new ones for multi-view and multi-domain ophthalmic applications.
  • Enhanced Understanding: It will facilitate a deeper understanding of disease progression and variations.
  • Addressing Bias: The dataset's diversity is expected to help in identifying and mitigating algorithmic bias.

Summary updated

Abstract

Data from clinical real-world settings is characterized by variability in quality, machine-type, setting, and source. One of the primary goals of medical computer vision is to develop and validate artificial intelligence (AI) based algorithms on real-world data enabling clinical translations. However, despite the exponential growth in AI based applications in healthcare, specifically in ophthalmology, translations to clinical settings remain challenging. Limited access to adequate and diverse real-world data inhibits the development and validation of translatable algorithms. In this paper, we present a new multi-modal longitudinal ophthalmic imaging dataset, the Illinois Ophthalmic Database Atlas (I-ODA), with the goal of advancing state-of-the-art computer vision applications in ophthalmology, and improving upon the translatable capacity of AI based applications across different clinical settings. We present the infrastructure employed to collect, annotate, and anonymize images from multiple sources, demonstrating the complexity of real-world retrospective data and its limitations. I-ODA includes 12 imaging modalities with a total of 3,668,649 ophthalmic images of 33,876 individuals from the Department of Ophthalmology and Visual Sciences at the Illinois Eye and Ear Infirmary of the University of Illinois Chicago (UIC) over the course of 12 years.
BibTeX
@inproceedings{mojab2021i,
  author = {Nooshin Mojab and Vahid Noroozi and Abdullah Aleem and Manoj Prabhakar Nallabothula and Joseph Baker and Dimitri T. Azar and Mark Rosenblatt and R. V. Paul Chan and Darvin Yi and Philip S. Yu and Joelle A. Hallak},
  title = {{I-ODA, Real-world Multi-modal Longitudinal Data for Ophthalmic Applications}},
  booktitle = {HEALTHINF},
  year = {2021},
  url = {https://openreview.net/forum?id=XsqHDC5Z2I},
}
Figure