Deep Learning based retinal layer segmentation in optical coherence tomography scans of patients with inherited retinal diseases

Franziska Eckardt, Robin Mittas, Nastassya Horlava, Johannes Schiefelbein, Ben Asani, Stylianos Michalakis, Maximilian Gerhardt, Claudia Priglinger, Daniel Keeser, Nikolaos Koutsouleris, Siegfried Priglinger, Fabian J. Theis, Tingying Peng, Benedikt SchwormmedRxiv 2023 · DOI

Abstract

On optical coherence tomography (OCT) scans of patients with inherited retinal diseases (IRDs), the outer nuclear layer (ONL) thickness measurement has been well established as a surrogate marker for photoreceptor preservation. Current automatic segmentation tools fail in OCT segmentation in IRDs, and manual segmentation is time consuming. We trained a U-net based model on healthy patients and applied a domain adaptation technique to IRD patients’ scans. We established an AI-based image segmentation algorithm that reliably segments the ONL in OCT scans of IRD patients. In a test dataset, the dice-score of the algorithm was 98.7%. Furthermore, we generated thickness maps of the full retinal thickness and the ONL layer for each patient. Accurate segmentation of anatomical layers on OCT scans plays a crucial role for predictive models linking retinal structure to visual function.

Introduction

Inherited Retinal Diseases (IRDs), such as Retinitis Pigmentosa, comprise a diverse group of genetic disorders that lead to progressive retinal degeneration and potential blindness. In clinical management and research, the thickness of the Outer Nuclear Layer (ONL) is a vital surrogate marker, as it directly reflects the density of surviving photoreceptors.

While Optical Coherence Tomography (OCT) allows for high-resolution visualization of these layers, a significant bottleneck exists: standard automated segmentation tools are optimized for healthy retinal architecture. In IRD patients, these layers are often pathologically thinned or structurally distorted, causing automated tools to fail. Consequently, clinicians must rely on manual segmentation, a process that is both labor-intensive and prone to human error. This study presents a deep learning-based approach to automate this process specifically for diseased retinal tissue.

Methodology

Architecture and Processing

The researchers developed a segmentation pipeline based on the U-Net architecture using TensorFlow. To account for the anisotropic resolution of OCT scans—where resolution between slices differs from resolution within a slice—the team utilized 2D convolutional kernels to process scans on a slice-by-slice basis.

Addressing Class Imbalance

A primary challenge in IRD segmentation is extreme class imbalance. In healthy eyes, the ONL accounts for approximately 4.0% of the total pixels; in IRD patients, this can drop to as little as 0.5%. To prevent the model from ignoring the thinned ONL, a weighted loss function was implemented, penalizing errors in the ONL four times more heavily than errors in other retinal layers.

Domain Adaptation

The "secret sauce" of this approach is the use of Adaptive Batch Normalization for domain adaptation. The model was first trained on a large dataset of healthy retinal scans. To adapt it to the unique structural characteristics of IRDs without requiring a massive new dataset, the researchers froze all trainable weights except for the Batch Normalization layers. This allowed the model to adjust to the "domain shift" of diseased tissue using only a small cohort of expert-annotated IRD scans.

Datasets

The study utilized two primary datasets captured via the Heidelberg Spectralis HRA+OCT platform:

  1. Healthy Controls: 18 volume scans from 12 healthy individuals, with annotations verified by experts.
  2. IRD Patients: 25 volume scans from 16 patients with confirmed IRD diagnoses. These were manually annotated by two expert ophthalmologists to establish a reliable ground truth.

Results and Discussion

The study evaluated several model iterations, with the domain adaptation approach (Model 4) proving most effective.

  • Performance Metrics: The model achieved an overall Dice score of 98.8% on the IRD test set. The ONL-specific Dice score reached 70.2%, a dramatic improvement over the 31.7% achieved by models trained only on healthy data.
  • The "ONL-hill" Challenge: The model successfully navigated the "ONL-hill," a convex thickening within the foveal pit that often confuses standard algorithms in diseased eyes.
  • Clinical Utility: The pipeline successfully generated full-retina and ONL-specific thickness maps (ETDRS grids), providing clinicians with a rapid, visual overview of disease progression that previously required hours of manual labor.

Conclusion and Future Work

This research demonstrates that deep learning, combined with targeted domain adaptation, can overcome the limitations of standard OCT segmentation tools in rare disease contexts. While the results are promising, the authors note that future work is needed to test the model on hardware from different manufacturers and to validate its performance on lower-quality scans often encountered in clinical practice. Ultimately, these automated measurements will be essential for correlating retinal structure with functional data in upcoming gene therapy trials.

Summary updated

Abstract

Abstract Background On optical coherence tomography (OCT) scans of patients with inherited retinal diseases (IRDs), the outer nuclear layer (ONL) thickness measurement has been well established as a surrogate marker for photoreceptor preservation. Current automatic segmentation tools fail in OCT segmentation in IRDs, and manual segmentation is time consuming. Methods and Material Patients with IRD and the availability of an OCT scan were screened for the present study. Additionally, OCT scans of patients without retinal disease were included, to provide training data for the artificial intelligence (AI). We trained a U-net based model on healthy patients and applied a domain adaption technique to IRD patients’ scans. Results We established an AI-based image segmentation algorithm that reliably segments the ONL in OCT scans of IRD patients. In a test dataset, the dice-score of the algorithm was 98.7% . Furthermore, we generated thickness maps of the full retinal thickness and the ONL layer for each patient. Conclusion Accurate segmentation of anatomical layers on OCT scans plays a crucial role for predictive models linking retinal structure to visual function. The here-presented OCT image segmentation algorithm could provide the basis for further studies on IRDs.
BibTeX
@misc{eckardt2023deep,
  author = {Franziska Eckardt and Robin Mittas and Nastassya Horlava and Johannes Schiefelbein and Ben Asani and Stylianos Michalakis and Maximilian Gerhardt and Claudia Priglinger and Daniel Keeser and Nikolaos Koutsouleris and Siegfried Priglinger and Fabian J. Theis and Tingying Peng and Benedikt Schworm},
  title = {{Deep Learning based retinal layer segmentation in optical coherence tomography scans of patients with inherited retinal diseases}},
  year = {2023},
  doi = {10.1101/2023.09.26.23295852},
  howpublished = {medRxiv},
}
Figure