Children's eyes can focus hard enough to fool the machines that measure them. When a child looks into an automated refractor, the focusing muscle inside the eye often contracts, and the eye reads as more nearsighted than it really is. This artifact is called pseudo-myopia. Clinics remove it with cycloplegic drops, which temporarily paralyze the focusing muscle. Mass school screenings skip the drops, and so they overcount myopia. In one cohort of 614 children, the undropped measurement put myopia prevalence at 76.7%, against a true 62.9% (Myopia Prediction Using Machine Learning: An External Validation Study). Machine-learning models now claim to predict the dropped ("cycloplegic") value from undropped measurements, and they report $R^2$ near 0.95 and mean absolute errors near 0.3 diopters (D).
This note argues that these headline numbers have the same weakness the language-model community found in contaminated or shortcut-prone benchmarks. Much of a high score can come from something other than the skill being claimed. The undropped refraction already contains most of the true refraction. A model that echoes its input plus a constant can therefore score high $R^2$ while learning nothing about how much a particular child is over-accommodating. On top of that, splits that place a child's two eyes, or several photos of the same patient, on both sides of the train/test line leak identity into the test set. We propose to re-score the existing correction models on the part of the problem that is actually hard: the per-child accommodation offset. We use grouped splits and artifact-detecting baselines borrowed from the benchmark-contamination literature, and we ask whether the published models beat trivial references once the easy part is removed.
Core Claim
Let $S_{nc}$ be a child's undropped spherical equivalent (sphere plus half the cylinder, in diopters) and $S_{c}$ the cycloplegic value. The quantity a correction model must supply is the offset
$$\Delta = S_{c} - S_{nc},$$
the amount of refraction hidden by active focusing. For any target $y$ with $\operatorname{Var}(\Delta) \ll \operatorname{Var}(S_{c})$, $R^2$ on the total refraction $S_c$ is dominated by the input itself. We therefore score models with a residual skill score
$$\text{Skill} = 1 - \frac{\text{MAE}_{\text{model}}(\Delta)}{\text{MAE}_{\text{ref}}(\Delta)},$$
where $\text{MAE}_{\text{ref}}$ is the error of a reference that predicts $\Delta$ from age alone, fitted on the training fold. Skill near zero means the biometric or lens features add nothing beyond "younger children accommodate more."
Our claim is that published accuracy should be reported on $\Delta$, under child-level and site/device-level grouping, against this reference. We expect a substantial fraction of the reported gain to disappear. Whatever remains is the model's real contribution, and it should concentrate in young children and in lens-curvature features. That would be the first honest estimate of how far computation can substitute for drops.
The benchmark-contamination literature is the methodological lens here, not a second application. Back to Square One motivates the approach. It showed that WinoGrande-style scores partly reflect artifacts: models scored over 60% with the answer candidates deleted, and top WSC scores fell from 89.71% to 79.41% under paired "group scoring." We reuse its two tools. The first is partial-input baselines, analogous to deleting the answer candidates. The second is a harder scoring rule that removes the easy component. DyVal showed that models strong on static benchmarks scored near zero on freshly generated items. It supplies the leakage-injection logic: deliberately contaminate a split and measure how far the score inflates.
Research Program
The work has three parts.
Part 1: Re-score the tabular correction models on the offset. The first part re-analyzes the two tabular cohorts in the file. The first is the 153-child cohort (ages 4–15) of Predictive Modeling of Cycloplegic Refraction Using Non-Cycloplegia Ocular Parameters With Emphasis on Lens-Related Features. That study used gradient-boosted trees with LASSO feature selection, adding lens features derived via the SRK/T formula, and reached MAE 0.307 D and $r^2$ 0.964. The second is the 614-child cohort of the external validation study. There, gradient-boosted trees predicting cycloplegic spherical equivalent reached $R^2$ 0.95 and MAE 0.32 D, across two drugs and two biometers.
Both papers report metrics on total refraction. Neither, as far as the file shows, reports error on $\Delta$ or a comparison against an input-plus-offset reference. The 614-cohort classifier's metrics are reported per eye as well as per person. We will need row-level data from the authors, or a comparable prospectively collected cohort. We state this dependence plainly: it is the main practical risk of the proposal.
Part 2: Audit the fundus-photo model for leakage and shortcuts. The second part audits the fundus-photograph refraction system of Identification of ocular refraction based on deep learning algorithm as a novel retinoscopy method and Identification of ocular refraction using a novel intelligent retinoscopy system. This system fuses two convolutional networks and reaches sphere MAE 0.50 D. Its data were 11,973 images from 3,954 patients split 70/20/10, and the file does not say whether the split was by image or by patient. The papers also report that adding age moved sphere MAE from 0.63 D to 0.50 D.
Here the undropped refraction is not an input, so the identity reference does not apply. We will instead run three checks. First, an age-only regressor, the partial-input baseline. Second, a patient-grouped versus image-level split comparison, the leakage test. Third, a camera and site hold-out where available, since the papers validated on a single desktop camera.
Part 3: A correction model trained on the offset. The third part builds a small model trained on $\Delta$ directly, with grouped cross-validation. It uses non-cycloplegic biometry, the lens anterior and posterior curvature radii, and age with a monotone constraint. The 153-child study found that the anterior lens curvature changes more than the posterior during accommodation, which gives a physiological reason to expect lens features to carry offset information that age cannot. This model tests whether a residual target helps. It is not meant as the main contribution.
Baselines
Do-nothing reference. The model predicts $S_c = S_{nc}$, so $\Delta = 0$. At the prevalence level this is the 76.7% apparent prevalence against the true 62.9% reported in the 614-child cohort. No reported MAE for this reference was found on Lacuna. It anchors how much any model helps at all.
Offset references. These predict $S_c = S_{nc} + \hat\Delta$, where $\hat\Delta$ is either the training-fold mean offset or a spline in age. They are fitted on exactly the same training folds as the learned models, with no tuning budget beyond choosing the spline degrees of freedom by inner cross-validation. They are the strongest cheap competitors because they capture the population-average pseudo-myopia and its known age dependence. Those are exactly the parts a skeptic suspects the published models mostly recover.
Strongest existing methods. These are the gradient-boosted tree models of the two tabular studies. The reported figures are MAE 0.32 D and $R^2$ 0.95 on total spherical equivalent (Myopia Prediction Using Machine Learning), and MAE 0.307 D and $r^2$ 0.964 (Predictive Modeling of Cycloplegic Refraction). For the image setting, the fused-network system reports sphere MAE 0.50 D (Identification of ocular refraction based on deep learning algorithm).
We will retrain each with its published feature sets and a matched hyperparameter search (the same number of configurations as our own model). The retraining uses child-grouped folds, so both eyes of a child always fall on the same side. It will be scored on $\text{MAE}(\Delta)$, on residual skill, and on absolute prevalence error at the myopia threshold.
A learned model "beats" the reference only if the 95% bootstrap interval of its residual skill, resampled over children, excludes zero. The same must hold within the preschool and early-school age bands, where the offset is largest and most variable.
Ablations. Two ablations isolate the key ingredient. For the tabular models, we remove the lens-curvature features, which tests whether they carry offset information beyond age. For the image model, we remove age fusion and compare image-level with patient-level splits. The size of that last gap is our direct measure of leakage inflation.
Why not just…?
Why not just trust external validation, which the 614-child study already did? External validation guards against overfitting to one clinic. It does not change the metric. A total-refraction $R^2$ of 0.95 is compatible with near-zero skill on the offset, because the input already explains most of the variance. We can also compute gaps from the prevalence figures in that paper. The models' predicted prevalences, 58.8% and 60.6%, sit 4.1 and 2.3 points from the truth. A single calibrated offset could plausibly close a similar gap. This is a real threat to the value of the published models, and the offset-reference comparison in Part 1 settles it directly. If the references match the trees on prevalence and on $\text{MAE}(\Delta)$, that finding is the result.
Why not just fix the splits to be child-level and leave the metric alone? Grouped splitting is necessary, and we adopt it. But it addresses only one of the two inflation sources. A child-grouped split still rewards input copying on the total-refraction target. The ablation separating split-level inflation from target-level inflation will show which source dominates. If grouping alone explains most of the drop, we will say the metric change matters less than we claim.
Why not just mandate cycloplegia, or run brand-new private cohorts? Drops are the gold standard, but their 30–40 minute onset, the light sensitivity they cause, and the staffing they require make them infeasible at registry scale, such as the 1.2-million-child registry in Prevalence Patterns and Onset Prediction of High Myopia, which uses undropped screening. Correction models exist precisely to serve that setting. A fresh private cohort, analogous to the unpublished exam of Evaluating Large Language Models in Ophthalmology, would remove leakage, but it would still be scored on the wrong target. Our protocol applies equally to such a cohort.
Why not just probe large language or vision-language models for pseudo-myopic bias instead? The existing ophthalmology benchmarks, OphthBench, EyecareGPT, and LMOD, omit refraction entirely. LMOD shows generalist models far below supervised baselines on imaging (anatomy F1 averaging 0.22). The deployed refraction tools are supervised models, and their evaluation is where inflated numbers reach clinical decisions. Probing generalist models is a reasonable follow-up. It is not the higher-stakes question.
Decisive Checks
The first check is the headline comparison. We compute residual skill of the published tree models against the age-spline offset reference, under child-grouped cross-validation, on the 153- and 614-child cohorts, stratified by age band. If skill is near zero outside the youngest band, the literature's accuracy claims describe input copying.
The second check is a leakage-injection experiment on the fundus data. We train with image-level and patient-level splits and report the sphere MAE difference. We also report the age-only regressor's MAE as the floor any image model must clear.
The third check is a label-definition hold-out on the 614-child cohort. We train on tropicamide-labeled children and test on cyclopentolate-labeled children, and likewise across the two biometers. Tropicamide is the weaker agent, so its "ground truth" may retain some accommodation. Transfer failure would show that the target itself is partly contaminated, not only the split.
A positive outcome would be residual skill reliably above zero in young children, carried mainly by lens-curvature features. That would give screening programs a defensible reason to use these models. A negative outcome would tell them that an age-calibrated offset does as well, at a fraction of the cost.
References
- Myopia Prediction Using Machine Learning: An External Validation Study
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Predictive Modeling of Cycloplegic Refraction Using Non-Cycloplegia Ocular Parameters With Emphasis on Lens-Related Features
- Identification of ocular refraction based on deep learning algorithm as a novel retinoscopy method
- Identification of ocular refraction using a novel intelligent retinoscopy system
- Prevalence Patterns and Onset Prediction of High Myopia for Children and Adolescents in Southern China via Real-World Screening Data: Retrospective School-Based Study
- Evaluating Large Language Models in Ophthalmology
- OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology
- EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Model
- LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models