Retest variability and patient reliability indices of quantitative fundus autofluorescence in age-related macular degeneration: a MACUSTAR study report

Leon von der Emde, Merten Mallwitz, Marc Vaisband, Jan Hasenauer, Marlene Saßmannshausen, Jan Henrik Terheyden, Hansjürgen Agostini, Lebriz Altay, R. Atia, F. Bandello, P. G. Basile, J. Batuca, Charlotte Behning, M. Belmouhand, Moritz Berger, Alison M. Binns, C. J. F. Boon, M. Böttger, C. Bouchet, John Brazier, Thomas Butt, C. Carapezzi, J. Carlton, Ângela Carneiro, A. Charil, R. Coimbra, Mariano Cozzi, David P. Crabb, José Cunha‐Vaz, C. Dahlke, Luís de Sisternes, H. Dunbar, Robert P. Finger, E. Fletcher, H. Floyd, C. Francisco, Matthias Gutfleisch, Sonja Hinz, Ruth Hogg, Frank G. Holz, C. B. Hoyng, A. Kilani, Jörn Krätzschmar, L. Kühlewein, Michael Larsen, Sérgio Leal, Yara Lechanteur, U. F. O. Luhmann, A. Lüning, I. Marques, C. Martinho, Giovanni Montesano, Z. Mulyukov, M. Paques, Maurízio Battaglia Parodi, Mariacristina Parravano, S. Penas, T. Peters, T. Peto, Maximilian Pfau, Stephen Poor, S. Priglinger, Donna Rowen, G. S. Rubin, José‐Alain Sahel, Carina C. Sanchez, O. Sander, Matthias Schmid, Steffen Schmitz-Valckenberg, J. Siedlecki, Rufino Silva, A Skelly, E. Souied, Giovanni Staurenghi, L. Stöhr, D. J. Taylor, Sarah Thiele, Adnan Tufail, M. Varano, L. Vieweg, Ludmila Wintergerst, Anna Wolf, N. Zakaria, Kenneth R. Sloan, Thomas AchScientific 2023 · DOI

Abstract

This study evaluates the retest variability of quantitative fundus autofluorescence (QAF) in patients with age-related macular degeneration (AMD) across a multicenter setting. Analyzing 132 eyes from 68 patients, the research demonstrates that QAF is a robust tool for monitoring disease progression. A key finding is that image quality, measured by Mean Opinion Scores (MOS), is the primary predictor of retest reliability. Furthermore, the study introduces an automated Random Forest Regressor (RF-MOS) that accurately predicts image quality, offering a path toward objective eligibility criteria in clinical trials.

Introduction: The Need for Reliable Biomarkers in AMD

Age-related macular degeneration (AMD) is a leading cause of vision loss, particularly in its early and intermediate stages where effective therapies are still lacking. Developing these treatments requires reliable "biomarkers"—measurable biological signs that can track disease progression or treatment response. Fundus autofluorescence (FAF) is widely used to visualize the health of the retinal pigment epithelium (RPE), but standard FAF is qualitative and difficult to compare across different clinics or time points.

Quantitative FAF (QAF) addresses this by using an internal scaling bar to normalize data. However, for QAF to be effective in large-scale clinical trials, its retest variability must be low enough to distinguish true biological changes from measurement "noise." This study investigates the reliability of QAF in a real-world, multicenter environment.

Methods and Data

The researchers utilized the MACUSTAR clinical study cohort, a large European multicenter project. The dataset included 132 eyes from 68 participants, covering a spectrum from healthy controls to late-stage AMD.

Imaging Protocol

Patients underwent duplicate imaging twice on the same day (intra-day) and again two weeks later (inter-day) across six different study sites. This rigorous protocol allowed for the assessment of both short-term and medium-term consistency.

Quality Assessment

Two experts graded image quality on a 5-point Mean Opinion Score (MOS) based on focus, illumination, and centering. To move toward automation, the team trained a Random Forest Regressor (RF-MOS) to predict these quality scores directly from raw image data.

Examples of QAF images with varying quality and their corresponding human (MOS) and automated (RF-MOS) scores.

Key Insights: Quality Over Pathology

The study's most significant finding was that a patient’s disease stage was not the primary hurdle for reliability. Instead, image quality (MOS) was the dominant factor influencing measurement variability. High-quality images yielded highly reliable data, regardless of whether the patient had AMD or was a healthy control.

The "Sweet Spot" and Automation

Reliability was highest in the middle ring of the retina (the QAF8 segments) compared to peripheral areas, which are more prone to shadowing. Crucially, the automated RF-MOS model performed remarkably well, predicting human quality scores with a mean absolute error of only 0.27.

Correlation between manual Mean Opinion Scores (True MOS) and automated Random Forest Regressor scores (RFR-MOS).

Results: High Consistency Across Sites

The results confirm that QAF is "ready for prime time" in large-scale trials:

  • High Consistency: For the primary region of interest (QAF8), the Intraclass Correlation Coefficient (ICC) was 0.95 for same-day repeats and 0.79 for 2-week repeats.
  • Reliability Boost: Applying a strict quality cutoff (MOS ≥ 4.5) improved inter-day reliability to 0.84.
  • Predictive Power: MOS (and its automated counterpart) was the only statistically significant predictor of retest variation in both intra-day and inter-day scenarios.

Discussion and Future Directions

While the study proves the multicenter viability of QAF, some limitations remain. The cohort had few "early AMD" patients, and detailed data on cataract severity—which can scatter light—was unavailable.

The findings suggest that future clinical trials should adopt automated quality metrics as objective eligibility criteria. By ensuring that only patients who can be imaged reliably are enrolled, researchers can increase the statistical power of studies to detect the therapeutic effects of new AMD treatments.

Summary updated

Abstract

This study aimed to determine the retest variability of quantitative fundus autofluorescence (QAF) in patients with and without age-related macular degeneration (AMD) and evaluate the predictive value of patient reliability indices on retest reliability. A total of 132 eyes from 68 patients were examined, including healthy individuals and those with various stages of AMD. Duplicate QAF imaging was conducted at baseline and 2 weeks later across six study sites. Intraclass correlation (ICC) analysis was used to evaluate the consistency of imaging, and mean opinion scores (MOS) of image quality were generated by two researchers. The contribution of MOS and other factors to retest variation was assessed using mixed-effect linear models. Additionally, a Random Forest Regressor was trained to evaluate the extent to which manual image grading of image quality could be replaced by automated assessment (inferred MOS). The results showed that ICC values were high for all QAF images, with slightly lower values in AMD-affected eyes. The average inter-day ICC was found to be 0.77 for QAF segments within the QAF8 ring and 0.74 for peripheral segments. Image quality was predicted with a mean absolute error of 0.27 on a 5-point scale, and of all evaluated reliability indices, MOS/inferred MOS proved most important. The findings suggest that QAF allows for reliable testing of autofluorescence levels at the posterior pole in patients with AMD in a multicenter, multioperator setting. Patient reliability indices could serve as eligibility criteria for clinical trials, helping identify patients with adequate retest reliability.
BibTeX
@inproceedings{emde2023retest,
  author = {Leon von der Emde and Merten Mallwitz and Marc Vaisband and Jan Hasenauer and Marlene Saßmannshausen and Jan Henrik Terheyden and Hansjürgen Agostini and Lebriz Altay and R. Atia and F. Bandello and P. G. Basile and J. Batuca and Charlotte Behning and M. Belmouhand and Moritz Berger and Alison M. Binns and C. J. F. Boon and M. Böttger and C. Bouchet and John Brazier and Thomas Butt and C. Carapezzi and J. Carlton and Ângela Carneiro and A. Charil and R. Coimbra and Mariano Cozzi and David P. Crabb and José Cunha‐Vaz and C. Dahlke and Luís de Sisternes and H. Dunbar and Robert P. Finger and E. Fletcher and H. Floyd and C. Francisco and Matthias Gutfleisch and Sonja Hinz and Ruth Hogg and Frank G. Holz and C. B. Hoyng and A. Kilani and Jörn Krätzschmar and L. Kühlewein and Michael Larsen and Sérgio Leal and Yara Lechanteur and U. F. O. Luhmann and A. Lüning and I. Marques and C. Martinho and Giovanni Montesano and Z. Mulyukov and M. Paques and Maurízio Battaglia Parodi and Mariacristina Parravano and S. Penas and T. Peters and T. Peto and Maximilian Pfau and Stephen Poor and S. Priglinger and Donna Rowen and G. S. Rubin and José‐Alain Sahel and Carina C. Sanchez and O. Sander and Marlene Saßmannshausen and Matthias Schmid and Steffen Schmitz-Valckenberg and J. Siedlecki and Rufino Silva and A Skelly and E. Souied and Giovanni Staurenghi and L. Stöhr and D. J. Taylor and Jan Henrik Terheyden and Sarah Thiele and Adnan Tufail and M. Varano and L. Vieweg and Ludmila Wintergerst and Anna Wolf and N. Zakaria and Kenneth R. Sloan and Steffen Schmitz-Valckenberg and Robert P. Finger and Frank G. Holz and Thomas Ach},
  title = {{Retest variability and patient reliability indices of quantitative fundus autofluorescence in age-related macular degeneration: a MACUSTAR study report}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-43417-y},
}
Figure