Abstract
Clinical diagnosis of the eye is performed over multifarious data modalities including scalar clinical labels, vectorized biomarkers, two-dimensional fundus images, and three-dimensional Optical Coherence Tomography (OCT) scans. While the clinical labels, fundus images and OCT scans are instrumental measurements, the vectorized biomarkers are interpreted attributes from the other measurements. Clinical practitioners use all these data modalities for diagnosing and treating eye diseases like Diabetic Retinopathy (DR) or Diabetic Macular Edema (DME). Enabling usage of machine learning algorithms within the ophthalmic medical domain requires research into the relationships and interactions between these relevant data modalities. Existing datasets are limited in that: (i) they view the problem as disease prediction without assessing biomarkers, and (ii) they do not consider the explicit relationship among all four data modalities over the treatment period. In this paper, we introduce the Ophthalmic Labels for Investigating Visual Eye Semantics (OLIVES) dataset that addresses the above limitations. This is the first OCT and fundus dataset that includes clinical labels, biomarker labels, and time-series patient treatment information from associated clinical trials. The dataset consists of 1268 fundus eye images each with 49 OCT scans, and 16 biomarkers, along with 3 clinical labels and a disease diagnosis of DR or DME. In total, there are 96 eyes' data averaged over a period of at least two years with each eye treated for an average of 66 weeks and 7 injections. OLIVES dataset has advantages in other fields of machine learning research including self-supervised learning as it provides alternate augmentation schemes that are medically grounded.
Introduction
In modern ophthalmology, clinical diagnosis is a multi-modal synthesis. Practitioners do not rely on a single image; instead, they integrate data from eye charts (clinical labels), 2D retinal photography (fundus), and 3D cross-sectional scans (OCT) to monitor disease progression and treatment response. Despite this, most existing machine learning datasets are "one-shot" snapshots focused narrowly on binary disease classification (e.g., presence or absence of Diabetic Retinopathy).
This narrow focus creates a significant gap: AI models often lack the context of how a disease manifests through specific biological indicators (biomarkers) or how these indicators evolve following clinical intervention. Furthermore, standard data augmentation techniques in computer vision, such as random cropping, can inadvertently destroy the fine-grained medical features essential for accurate diagnosis. The OLIVES dataset is designed to provide the "connective tissue" between these disparate modalities and the temporal dimension of clinical treatment.
The OLIVES Dataset
The Ophthalmic Labels for Investigating Visual Eye Semantics (OLIVES) dataset is derived from two major clinical trials: PRIME and TREX DME. It represents a significant leap in scale and depth for ophthalmic research:
- Scale: The dataset includes 78,189 OCT slices and 1,268 fundus images.
- Comprehensive Labeling: It features 16 unique biomarkers (such as intra-retinal fluid and membrane disruption), 4 clinical labels (including Visual Acuity and Retinal Thickness), and longitudinal data for 96 eyes over an average of 66 weeks.
- Longitudinal Context: Unlike popular static datasets like Kermany or Kaggle DR, OLIVES provides multi-modal pairings and treatment-over-time data, allowing for the study of disease dynamics.
Methodology and Research Directions
The researchers established benchmarks for three core machine learning paradigms using the OLIVES dataset:
- Multi-Modal Integration: Utilizing a "Guided Loss" function, models are trained where features from one modality (e.g., clinical labels) optimize the learning of another (e.g., OCT scans). This allows standard architectures like ResNet-18 to incorporate expert clinical knowledge.
- Medically-Grounded Contrastive Learning: Rather than using generic image augmentations, the authors use clinical metadata (like Patient ID or Visual Acuity) to define "positive pairs" for self-supervised learning. Scans from the same eye are treated as similar, forcing the model to learn stable anatomical features rather than noise.
- Time-Series Analysis: The dataset enables benchmarking the ability of architectures (ResNet, DenseNet, EfficientNet, and ViT) to predict future vision outcomes based on current longitudinal scans.
Key Insights and Results
Biomarkers as Intermediate Representations
A primary finding of the study is that biomarkers are significantly more effective features for disease detection than raw images alone. Utilizing biomarkers as an intermediate representation led to a nearly 10% boost in classification accuracy, achieving a Balanced Accuracy of 82.33% for DR/DME detection compared to 70.15% using OCT scans in isolation.
Clinical Supervision in Self-Supervised Learning
Using clinical data, such as Central Subfield Thickness (CST), to guide contrastive learning outperformed state-of-the-art unsupervised methods like SimCLR. Specifically, CST-guided learning yielded an AUROC of 0.790, surpassing standard unsupervised PCL (0.767). This suggests that medical metadata is a powerful, often untapped source of supervision.
Treatment-Induced Domain Shift
The research demonstrates that medical treatments cause a "domain shift" in the data. Models trained on initial patient visits may struggle with scans taken after several rounds of injections because the medication alters the underlying anatomy. Predicting visit-to-visit improvement remains a challenge, with EfficientNet achieving a peak accuracy of 60.65%, highlighting the complexity of biological response prediction.
Conclusion and Future Work
While the OLIVES dataset provides a robust foundation for multi-modal ophthalmic research, it is currently limited by its source from a single clinical site and the absence of a "healthy" control group. Future work will focus on expanding the dataset to more diverse populations and exploring active learning strategies to identify which clinical labels are most valuable to collect, potentially reducing the diagnostic burden on medical professionals.
Summary updated