Older adults tend to look at the centre of objects and scenes and to skip the fine detail at the edges. People with early cognitive decline appear to do the same thing, only more so. This overlap is a problem for any screening test built on where people look. A tablet that records a patient's gaze while they describe a picture could flag "centralized, coarse viewing," but that signal is mostly a measure of age. We propose to measure something else: what happens right after a standardized spoken question asks the person about a specific fine peripheral detail. A healthy 78-year-old who habitually looks at the centre should, when asked, move their eyes to the detail, find it, and name it. A person whose decline lies in processing rather than habit should arrive late, fail to settle on the detail, or look at it without being able to report it.
The test we propose is a short picture-description interview. Gaze is tracked with the tablet's front camera, and speech is recorded at the same time. Every participant hears the same fixed sequence of prompts. Half of the prompts target fine-scale peripheral details and half target coarse central content. The diagnostic quantity is the recovery that the prompt produces, compared with how the person was viewing beforehand and with age-matched norms. The raw viewing pattern is not the signal. The design addresses a known weakness of conversational screening, which is that interviewer behaviour contaminates speech markers. It also addresses a known weakness of gaze screening, which is that it confuses normal aging with disease.
Core Claim
Our claim is that a prompted scale shift separates cognitive impairment from age better than either speech markers alone or passive gaze statistics. The measure has three parts: the latency, the accuracy of eye landing, and the accuracy of the spoken report after a scripted question about a fine peripheral detail. It is compared with the same measure for coarse central targets, which serve as a within-person control for general slowing.
Specifically, we predict two things. In healthy controls, passive centre bias will rise with age but prompted recovery will stay roughly flat. In mild cognitive impairment (MCI), recovery will fail even after controlling for age. We also predict that adding recovery features raises discrimination of MCI from controls above a strong speech-only model on the same recordings.
Why this and not the obvious combination
The obvious idea at this intersection is to put commodity eye tracking into a Cookie Theft description and feed gaze plus speech into a classifier. The file shows nobody has done exactly that. It also suggests the result would be hard to interpret.
Several lines of work motivate the proposal by showing that gaze scale shifts with age and with decline:
- Gaze distribution analysis and saliency prediction across age groups fitted an age-adapted saliency model with a variable centre-bias weight and age-dependent feature scale. It works for children through young adults, and extending it to older or impaired adults is left open.
- Gaze Fixation Patterns Correlate with Visual Attention and Memory found that adults aged 50 and over fixate object centres rather than rims and make far more pattern-separation errors (31.3% vs. 8.3%).
- Eye-movement patterns in face recognition are associated with cognitive decline in older adults showed that a holistic, centrally fixated face-viewing strategy correlates with lower Montreal Cognitive Assessment (MoCA) scores (r = −0.48 and −0.44), independent of age.
All three measure spontaneous viewing on static lab tasks. None asks whether the pattern can be reversed on request.
Two papers constrain the design:
- Exploring emotional expression recognition in aging adults using the Moving Window Technique masked the visual field and found that aging deficits reflect processing, not just fixation location. So a fixation map alone cannot be the biomarker. What matters is whether information is extracted once the eyes arrive, and that is exactly what a prompt followed by a spoken report tests.
- “Is There Anything Else?” showed that interviewer turns change pronoun and type-token markers and break transfer from the Pitt corpus to the Wisconsin Longitudinal Study corpus. So any prompt we add must be scripted, timed by rule, and logged, not improvised.
The closest multimodal systems do not measure gaze:
- Towards Scalable Remote Assessment of Mild Cognitive Impairment Via Multimodal Dialog runs a virtual agent that extracts speech, facial landmarks and self-reported symptoms, but no gaze.
- Cognitive Insights Across Languages analyses picture-description interviews from speech and text only.
- Analyzing Attention Focus in the Cookie Theft Picture Description Task Using Word Alignment infers attention from word timestamps aligned to image regions. It names the absence of real eye tracking as its main limitation.
Our proposal differs from all of these in making the probe–response contrast, not the passive recording, the object of measurement.
Research Program
Stimuli. We will build a stimulus set of the Cookie Theft drawing plus eight colour naturalistic scenes. Each scene has annotated regions at two scales: central gist regions (people, main action) and fine peripheral targets (small objects near edges, texture or rim details distinguishable only at high spatial frequency). For each scene we also build a matched lure image that differs only in one fine peripheral target. This gives an embedded pattern-separation check in the style of the ETAM protocol from Danilov et al.
Interview script. The interview is semi-structured in the sense of Murray et al.: a fixed guide with open prompts. We remove wording freedom and keep only ordering rules. Each scene has three stages:
- A free description opens with a single standardized instruction.
- A synthesized voice then delivers four scripted probes. Two are fine-peripheral ("What is on the shelf in the top left corner?") and two are coarse-central. The order is counterbalanced.
- A closing recognition trial asks whether the scene or the lure was seen.
Timing follows a fixed rule, so the "administrator" is identical for everyone.
Acquisition. Gaze is recorded on a tablet with the front-camera method of Accelerating eye movement research via accurate and affordable smartphone eye tracking. That work reported 0.46 ± 0.03 cm error after a roughly 30-second calibration, with saliency heatmaps agreeing with a Tobii system (object-level r = 0.90). Eye Tracking for Everyone serves as the uncalibrated fallback, with 1.34 cm calibrated error on phones. Audio is recorded in sync.
Features. For each probe $p$ we compute a recovery score:
$$R_p = \frac{\mathbb{1}[\text{verbal report correct}]}{1 + t_p / \tau},$$
Here $t_p$ is the time from the end of the probe to the first fixation within the target region. $\tau$ is the median of $t_p$ in the youngest control group. The score is zero if the target is never fixated or the report is wrong. From these we form three quantities:
- Fine-scale recovery: the difference between the mean of $R_p$ over fine probes and the mean over coarse probes. Subtracting the coarse probes removes general motor, lexical and oculomotor slowing.
- Look-but-not-report rate: the fraction of probes where the target was fixated but the answer was wrong. This operationalizes the processing-vs-fixation dissociation from Birmingham et al.
- Pre-probe centre bias and entropy: passive viewing statistics that act as covariates.
Age norms. We extend the age-adapted saliency model of Krishna et al. (also Gaze distribution analysis and saliency prediction across age groups) to ages 50–85 by fitting its centre-bias weight and scale selection as smooth functions of age on controls only. Every passive gaze statistic is then expressed as a deviation from the participant's age-predicted map.
Cohort. We plan about 240 participants aged 50–85: roughly 120 cognitively healthy and 120 with clinically diagnosed MCI. All have MoCA and a trail-making test so the results can be compared directly with Chan et al. Recruitment is stratified by age decade, and the whole cohort returns at 12 months. A 40-person subset is recorded simultaneously with a lab eye tracker to validate tablet gaze in older eyes. Valliappan et al.'s accuracy figures are not reported by age in the file, so this validation has to be done rather than assumed.
Baselines
Naive reference. Age plus education in a logistic regression. This is the "do nothing" point. It matters here because both passive gaze and speech carry age information: Spoken language biomarkers for detecting cognitive impairment found demographics alone gave 0.79 AUC on the Framingham cohort. Any feature set that does not clearly beat this is mostly re-encoding age.
Strongest speech-only method. The ADReSSo-style late fusion of acoustic features (the eGeMAPS acoustic feature set with Active Data Representation) and automatic speech recognition (ASR) transcript features from Detecting Cognitive Decline Using Speech Only: The ADReSSo Challenge. That paper reports 78.87% test accuracy for AD classification and 68.75% accuracy (F1 66.67%) for two-year decline. We supplement it with the Whisper-plus-BERT-plus-OpenSMILE fusion of Cognitive Insights Across Languages, which reports 75.09 ± 1.8% unweighted average recall (UAR) in cross-validation and 56.18% on the held-out TAUKADIAL test. We also include the lightweight classical pipeline of Who needs context?, since small cohorts favour it.
These are the strongest competitors because they need no extra hardware and are well tuned for picture-description speech. We will run all three on our recordings, using the free-description plus probe-answer audio. Nested cross-validation, identical folds, and the same hyperparameter search budget will be used for every model. First we will reproduce the ADReSSo pipeline on ADReSSo data, to confirm the implementation before transfer. Beating this baseline means the full model's AUC exceeds the best speech-only AUC by a margin that is significant under a DeLong test on the same participants, both at baseline and on the 12-month decline label.
Passive-gaze multimodal system. The same speech features plus pre-probe passive gaze statistics: centre bias, entropy, and an analytic-vs-holistic score computed with the hidden Markov model clustering of Chan et al. This represents the "obvious combination." Roesler et al. reports a 0.75 AUC for its facial-landmark dialog system on a different cohort. That number is only a rough reference, because the populations differ.
Speech-timestamp attention proxy. The word-to-region alignment features of Favaro et al.: time to first region mention, region visit counts, and transition-time variability. These are computed from the same audio. No classification number for this method is reported in the file (no reported result found on Lacuna for Favaro et al.'s AoI features as a classifier), so we will establish it ourselves.
Key ablations. Two ablations isolate the proposal's central ingredient. The first removes the coarse-probe subtraction, keeping only raw fine-probe recovery. The second removes the verbal-correctness term, keeping only gaze latency. If either version matches the full index, the "prompted, within-person, report-verified" framing is not doing the work we claim.
Why not just…?
Why not just use speech alone? This is the most serious threat. Speech-only models are cheap and sometimes very strong: Alhanai et al. reached 0.92 AUC on Framingham. However, performance falls sharply under ASR and new cohorts. ADReSSo worked with transcripts at roughly 60% word error rate. The INESC-ID system on ADReSS dropped to 54.17% test accuracy for acoustics despite 72.73% on development data. Cognitive Insights fell from 75.09% cross-validated UAR to 56.18% on the held-out test. Speech also cannot separate a person who did not see a detail from one who saw it but could not name it.
We do not assume gaze wins. The incremental-AUC test against the strongest speech pipeline, on identical participants, is the primary outcome. If recovery features add nothing, we report that result.
Why not just infer attention from word timestamps? It needs no camera, and Favaro et al. found real group differences with it. But a delay between looking at something and naming it is exactly where word-finding blocks sit, which Barrera-Altuna et al. ( The Interspeech 2024 TAUKADIAL Challenge: Multilingual Mild Cognitive Impairment Detection with Multimodal Approach) tie to lexical retrieval rather than visual search. Our synchronized recordings let us measure directly how often the first mention of a region follows its first fixation, and by how much, in each diagnostic group. If the proxy tracks real gaze closely and matches it for discrimination, the camera is unnecessary, and we would say so.
Why not just record passive gaze, with no probes? Passive centralization is what Chan et al. and Danilov et al. already showed correlates with decline, and it is simpler to collect. Its weakness is that it is tied to age, which our first decisive check tests directly. The passive-gaze baseline above is run on the same data. If passive statistics, once expressed as deviations from the age-adapted saliency norm, match the recovery index, then probes are unnecessary and the proposal reduces to age normalization.
Why not make the probes adaptive, triggered in real time when gaze is stuck centrally? We considered this, since it sounds like a natural fit for semi-structured interviewing. We rejected it as the core design. Trigger timing would then differ across people, which recreates the administrator confound from "Is There Anything Else?". A fixed script with gaze-conditioned analysis recovers most of the information, because we know after the fact whether the target had already been fixated. We keep a small counterbalanced gaze-contingent arm, about 40 participants, to test whether contingency adds signal beyond the fixed script.
Proposed Experiments and Decisive Checks
Hardware validity in older eyes. In the dual-recorded subset, tablet-vs-lab fixation-to-region agreement must be high enough that probe latencies correlate strongly across the two devices. If tablet error in adults over 70 is large relative to the size of the fine targets, the targets are enlarged before the main study.
Age dissociation. Among controls only, we regress pre-probe centre bias and fine-scale recovery on age. The proposal predicts that the age slope for centre bias is reliably positive, as extrapolated from Krishna et al.'s monotone trends and from Danilov et al. It also predicts that the recovery slope is near zero and substantially smaller. If recovery declines with age as steeply as centre bias does, the core claim fails.
Discrimination beyond age and speech. With nested cross-validation, we compare AUC for MCI vs. control across the feature sets described above: age-only, speech-only, passive gaze plus speech, timestamp proxy plus speech, and full. We also report the partial correlation of fine-scale recovery with MoCA controlling for age. Chan et al.'s passive r of −0.48 serves as the passive reference point.
Processing vs. location. The look-but-not-report rate should be higher in MCI than in controls, and it should predict errors on the lure-recognition trial. This would bring the ETAM pattern-separation effect into a conversational setting.
Longitudinal check. At 12 months, we test whether baseline recovery predicts MoCA change better than baseline speech features. The only numeric anchor in the file for speech is ADReSSo's 68.75% decline accuracy, which comes from a different cohort and a two-year window, so the comparison is qualitative.
Cross-stimulus transfer. Models trained on the colour scenes are tested on Cookie Theft, and the reverse. This asks whether the recovery index avoids the corpus-bound degradation reported for linguistic markers.
References
- Gaze distribution analysis and saliency prediction across age groups
- Gaze Fixation Patterns Correlate with Visual Attention and Memory: the Results of a Pilot Study in Healthy Subjects
- Eye-movement patterns in face recognition are associated with cognitive decline in older adults
- Exploring emotional expression recognition in aging adults using the Moving Window Technique
- “Is There Anything Else?”: Examining Administrator Influence on Linguistic Features from the Cookie Theft Picture Description Cognitive Test
- Towards Scalable Remote Assessment of Mild Cognitive Impairment Via Multimodal Dialog
- Cognitive Insights Across Languages: Enhancing Multimodal Interview Analysis
- Analyzing Attention Focus in the Cookie TheftPicture Description Task Using Word Alignment
- 'It could have been a lot worse': the psychological effects of farm-related serious injury in Victoria
- Accelerating eye movement research via accurate and affordable smartphone eye tracking
- Eye Tracking for Everyone
- Gaze distribution analysis and saliency prediction across age groups
- Spoken Language Biomarkers for Detecting Cognitive Impairment
- Detecting Cognitive Decline Using Speech Only: The ADReSSo Challenge
- Who needs context? Classical techniques for Alzheimer's disease detection
- The Interspeech 2024 TAUKADIAL Challenge: Multilingual Mild Cognitive Impairment Detection with Multimodal Approach
- The Interspeech 2024 TAUKADIAL Challenge: Multilingual Mild Cognitive Impairment Detection with Multimodal Approach